How to synchronize your Liveblocks Yjs document data to a PlanetScale MySQL database

Liveblocks allows you to build collaborative applications with Yjs, which is tightly integrated into our infrastructure. Using our webhooks and REST API, you can then retrieve the Yjs document data as it changes, and store it in your database.

What we’re building

In this guide we’ll be linking a Yjs application up to a PlanetScale MySQL database so that Yjs document data is automatically synchronized. This is enabled through the following:

• YDocUpdated webhook event
• Get Yjs Document REST API

This specific webhook is throttled

Note that the YDocUpdated webhook event is throttled at a rate of once every 5 seconds. This is because Yjs can update up to 60 times per second, and it would be impractical to run the webhook this frequently.

Create an endpoint in your project

When a webhook event is triggered, it can send a POST request to the back end in your project, and from within there we can update the database. In this guide, we’ll be using a Next.js route handler (API endpoint) as an example, but other frameworks work similarly.

In order to use webhooks, we’ll need to retrieve the headers and body from the request. Here’s the basic endpoint we’ll be starting from:

export async function POST(request: Request) {  const body = await request.json();  const headers = request.headers;
  // Handle webhooks and database  // ...
  return new Response(null, { status: 200 });}

Create this endpoint in your project, and make it available on localhost at the following URL:

/api/liveblocks-database-sync

Testing webhooks locally

Running webhooks locally can be difficult, but one way to do this is to use a tool such as localtunnel or ngrok which allow you to temporarily put your localhost server online.

If your project is running on localhost:3000, you can run the following command to generate a temporary URL that’s available while your localhost server is running:

$npx localtunnel --port 3000

localtunnel generates a base URL that can be placed into the Liveblocks webhooks dashboard for quick testing. To use this, take the full address of your webhook endpoint, and replace the domain in your localhost address with the generated URL.

# Take your local URLhttp://localhost:3000/api/liveblocks-database-sync
# Replace localhost with the generated domain, then copy ithttps://my-localtunnel-url.loca.lt/api/liveblocks-database-sync

You now have a URL that can be used in the webhooks dashboard.

Set up webhooks on the Liveblocks dashboard

To use webhooks, you need to pass your endpoint URL to the webhooks dashboard inside your Liveblocks project, and tell the webhook to trigger when Yjs document data has changed.

1. Select your project

   From the Liveblocks dashboard, navigate to the project you’d like to use with webhooks, or create a new project.

   

2. Go to the webhooks dashboard

   Click on the “Webhooks” tab on the menu at the left.

   

3. Create an endpoint

   Click the “Create endpoint…” button on the webhooks dashboard to start setting up your webhook.

   

4. Add your endpoint URL

   Enter the URL of the endpoint. In a production app this will be the real endpoint, but for now enter your localtunnel URL from earlier.

   

5. Get your secret key

   Click “Create endpoint” at the bottom, then find your “Webhook secret key” on the next page, and copy it.

   

6. Webhooks dashboard is set up!

   Note that you can filter specifically for ydocUpdated events, but we’re ignoring this for now so we can test more easily. Let’s go back to the code.

Verify the webhook request

The @liveblocks/node package provides you with a function that verifies whether the current request is a real webhook request from Liveblocks. You can set this up by setting up a WebhookHandler and running verifyRequest.

Make sure to add your “Signing Secret” from the Liveblocks dashboard—in a real project we’d recommend using an environment variable for this.

import { WebhookHandler } from "@liveblocks/node";
// Add your signing key from a project's webhooks dashboardconst WEBHOOK_SECRET = "YOUR_SIGNING_SECRET";const webhookHandler = new WebhookHandler(WEBHOOK_SECRET);
export async function POST(request: Request) {  const body = await request.json();  const headers = request.headers;
  // Verify if this is a real webhook request  let event;  try {    event = webhookHandler.verifyRequest({      headers: headers,      rawBody: JSON.stringify(body),    });  } catch (err) {    console.error(err);    return new Response("Could not verify webhook call", { status: 400 });  }
  // Update database  // ...
  return new Response(null, { status: 200 });}

We can then check we’re receiving the correct type of event, get the updated roomId, and handle updating the database inside there.

import { WebhookHandler } from "@liveblocks/node";
// Add your signing key from a project's webhooks dashboardconst WEBHOOK_SECRET = "YOUR_SIGNING_SECRET";const webhookHandler = new WebhookHandler(WEBHOOK_SECRET);
export async function POST(request: Request) {  const body = await request.json();  const headers = request.headers;
  // Verify if this is a real webhook request  let event;  try {    event = webhookHandler.verifyRequest({      headers: headers,      rawBody: JSON.stringify(body),    });  } catch (err) {    console.error(err);    return new Response("Could not verify webhook call", { status: 400 });  }
  // When Yjs document data has been updated  if (event.type === "ydocUpdated") {    const { roomId } = event.data;
    // Update database    // ...  }
  return new Response(null, { status: 200 });}

Get the current room’s Yjs document data

Before updating our database, we need to get the current room’s data. We can do this through the Get Yjs Document REST API. You use the REST API, you need to add your secret key from your project page.

import { WebhookHandler } from "@liveblocks/node";
// Add your signing key from a project's webhooks dashboardconst WEBHOOK_SECRET = "YOUR_SIGNING_SECRET";const webhookHandler = new WebhookHandler(WEBHOOK_SECRET);
// Add your secret key from a project's API keys dashboardconst API_SECRET = "sk_prod_xxxx…xxxxxxsk_prod_xxxxxxxxxxxxxxxxxxxxxxxx";
export async function POST(request: Request) {  const body = await request.json();  const headers = request.headers;
  // Verify if this is a real webhook request  let event;  try {    event = webhookHandler.verifyRequest({      headers: headers,      rawBody: JSON.stringify(body),    });  } catch (err) {    console.error(err);    return new Response("Could not verify webhook call", { status: 400 });  }
  // When Yjs document data has been updated  if (event.type === "ydocUpdated") {    const { roomId } = event.data;
    // Get Yjs data from Liveblocks REST API    const url = `https://api.liveblocks.io/v2/rooms/${roomId}/ydoc`;    const response = await fetch(url, {      headers: { Authorization: `Bearer ${API_SECRET}` },    });
    if (!response.ok) {      return new Response("Problem accessing Liveblocks REST APIs", {        status: 500,      });    }
    // Your JSON Yjs document data as a string    const yDocData = await response.text();
    // Update database    // ...  }
  return new Response(null, { status: 200 });}

Create a PlanetScale MySQL database

We’re ready to set up our PlanetScale database! We’ll be creating a simple documents table that contains the following fields:

1. Create a database

   Navigate to PlanetScale’s new database page, give your database a name, and click “Create” at the bottom.

   

2. Create a new branch

   After the database has been created, click “New branch” at the top right to create a branch—this works like Git.

   

3. Set up your schema

   Click “Console” at the top, select your branch, and enter the following to set up your table.

   CREATE TABLE documents (   roomId VARCHAR(255) PRIMARY KEY,   yDocData TEXT NOT NULL);

   

4. Click “Connect”

   After the database has been created, click “Connect” at the top right to create new credentials.

   

5. Create credentials

   Give your new credentials a name, then click “Create Password”.

   

6. Add the credentials to your project

   Select “Node.js” in the dropdown and copy the database URL (or add it as an environment variable).

   

7. Database ready!

   Let’s take a look at the code.

Add the Yjs data to your database

And finally, we can add the Yjs JSON data to our database! First, we need to install mysql2:

$npm i mysql2

Then implement the following to synchronize your data to PlanetScale:

import { WebhookHandler } from "@liveblocks/node";import mysql from "mysql2";
// Add your signing key from a project's webhooks dashboardconst WEBHOOK_SECRET = "YOUR_SIGNING_SECRET";const webhookHandler = new WebhookHandler(WEBHOOK_SECRET);
// Add your secret key from a project's API keys dashboardconst API_SECRET = "sk_prod_xxxx…xxxxxxsk_prod_xxxxxxxxxxxxxxxxxxxxxxxx";
// Your PlanetScale database URLconst DATABASE_URL = "YOUR_DATABASE_URL";
export async function POST(request: Request) {  const body = await request.json();  const headers = request.headers;
  // Verify if this is a real webhook request  let event;  try {    event = webhookHandler.verifyRequest({      headers: headers,      rawBody: JSON.stringify(body),    });  } catch (err) {    console.error(err);    return new Response("Could not verify webhook call", { status: 400 });  }
  // When Yjs document data has been updated  if (event.type === "ydocUpdated") {    const { roomId } = event.data;
    // Get Yjs data from Liveblocks REST API    const url = `https://api.liveblocks.io/v2/rooms/${roomId}/ydoc`;    const response = await fetch(url, {      headers: { Authorization: `Bearer ${API_SECRET}` },    });
    if (!response.ok) {      return new Response("Problem accessing Liveblocks REST APIs", {        status: 500,      });    }
    // Your JSON Yjs document data as a string    const yDocData = await response.text();
    // Update database    const connection = await mysql.createConnection(DATABASE_URL);
    const sql = `      INSERT INTO documents (roomId, yDocData)      VALUES (?, ?)      ON DUPLICATE KEY UPDATE          yDocData = VALUES(yDocData);    `;
    try {      await connection.query(sql, [roomId, yDocData]);    } catch (err) {      return new Response("Problem inserting data into database", {        status: 500,      });    }
    await connection.end();  }
  return new Response(null, { status: 200 });}

Check if it worked

To check if your database synchronization is working, you can replay a yDocUpdated event from the Liveblocks dashboard.

Then go back to the Console page on PlanetScale and enter the following to see all entries:

SELECT * FROM documents;

You should now see your Yjs document—we’ve successfully set up data synchronization! When a user edits Yjs data in your app, this function will be called, and your database will be updated. You can rely on this to stay up to date, within the 5 second throttle limit.

Learn more

You can learn more about Liveblocks webhooks in our full webhooks guide. We also have an API reference for the Get Yjs Document API.