Drizzle Team•2y ago

Share migrations in monorepo

I am using drizzle in a monorepo (npm workspaces with turborepo) within an internal package that is shared among multiple apps. It basically exports a bunch of functions that use drizzle behind the scenes. This works great, however I can't run migrations from within the apps that use the package, because the folder with the migrations is only available inside the package that contains drizzle. I would love to run the migrations on app startup using migrate from drizzle-orm/postgres-js/migrator like shown in the examples, but then I get the error that the migrations can't be found. The only solution I figured so far is to copy the migrations from the shared package to every app directory every time when I ran drizzle-kit generate. Can you think of any better solutions?

8 Replies

Andrii Sherman•2y ago

Hey, just say your message on stackoverflow and then remembered there was a help topic here, but just didn't have a chance to answer and now I found you Let's chat here I guess You can run drizzle-kit generate regardless of the structure you'll have. It can be a separate module that has drizzle-kit dependencies and uses a schema file plus an output folder to store migrations. The best approach would be to bundle the migrations folder in an app that executes migrations on app startup. I don't think you need to copy them to every package you have. However, if you don't have a "main" package, then I suppose each subpackage in the monorepo should bundle the migrations folder and run migrations on each app startup. I can provide more assistance if I can see some basic setup, perhaps a minimal reproducible example repository, so I can suggest some changes in a PR or something.

benjiman86OP•2y ago

Hey Andres, thanks for your time and the response - I really appreciate this. I made a really simple example repo to illustrate my question: https://github.com/BenBestmann/drizzle-monorepo The README contains more explanations. The main idea is that I want to encapsulate all database and business logic in an internal packages like the core package of the example repo and then use this package in multiple apps of the repo (e.g. web, mobile, backend), see the example Next.js app included (esp. page.tsx file). The approach works fine, except for the migrations. I would love to execute them in every app on startup like you illustrate in the docs because I find it very convenient, esp. in development. However as the core package is just an internal npm workspace with no build, when I execute the Client.migrate() function from within the apps they are looking for the migrations directory and of course can't find them. It somehow feels wrong to copy the migrations into every app, as I want the core package to be the single source of truth where I define all schema. Another alternative I can think of is to create a simple npm script in the core package that runs the migrations which I then manually inovke in development and via a CI solution during deployments. Do you have any other ideas? I also can think of using your push command, but we are using Postgres. Thanks again for taking the time! Best regards, Ben.

GitHub

GitHub - BenBestmann/drizzle-monorepo

Contribute to BenBestmann/drizzle-monorepo development by creating an account on GitHub.

benjiman86OP•2y ago

Sorry, I of course ment to type Andrew not Andres 😉

Andrii Sherman•2y ago

If my last name was San it actually may fit well San Andres It's a bit late here but I can take a look tomorrow and will try to help you

benjiman86OP•2y ago

Of course, thanks! Looking forward to your ideas.

Andrii Sherman•2y ago

Ok, so what you need to do is to have only 1 package responsible for migrations. As long as this package is core and not backend, you'll need to go with the CI approach and run migrations in CI right before server start. Another solution would be to make the backend responsible for migrations, and only the backend, because neither the web nor mobile should be responsible for it. So let's say you can have a core package containing all schemas and database calls, but generate migrations directly to the backend folder.

Copy code
export default {
  schema: "schema.ts",
  out: "../backend/migrations",
} satisfies Config;

Copy code
export default {
  schema: "schema.ts",
  out: "../backend/migrations",
} satisfies Config;

In this case, you'll still have a source of truth for all types in the core package, but SQL files generated from the kit will "live" in the backend and be executed when the backend starts. This means that if any new migration was created, you will need to redeploy the backend module.

benjiman86OP•2y ago

Thanks man, this helps. I like the idea with exporting the migration files to the backend. Nevertheless I think I will keep everything inside the core package, as it just feels more consistent. I will run the migrations via CI on deploy and during dev I will just use sth. like nodemon to automatically run migrations. drizzle-kit does not have a command yet to execute migrations, right? So I guess I will just write a little script to execute migrate() then? Also really looking forward for the release of drizzle-kit push for PostgresSQL. Thanks again for taking the time and your support! Drizzle is an awesome idea and I hope and guess you will habe great success with it! Defintely like it so far ✌️

agrattan•16mo ago

If anyone comes here looking for an example solution, this is what I got: migrate.ts in my database package

import { drizzle } from "drizzle-orm/postgres-js";
import { migrate } from "drizzle-orm/postgres-js/migrator";
import dotenv from "dotenv";
import postgres from "postgres";

// Load .env file before importing db
dotenv.config();

const client = postgres(
  process.env.DATABASE_URL ?? ""
);

const migrationDB = drizzle(client);

async function runDatabaseMigrations() {
  try {
    await migrate(migrationDB, { migrationsFolder: "./drizzle" });
    console.log("Migration completed! ✅");
    process.exit();
  } catch (error) {
    console.error("Migration failed! ❌");
    console.error(error);
  }
}

runDatabaseMigrations();

import { drizzle } from "drizzle-orm/postgres-js";
import { migrate } from "drizzle-orm/postgres-js/migrator";
import dotenv from "dotenv";
import postgres from "postgres";

// Load .env file before importing db
dotenv.config();

const client = postgres(
  process.env.DATABASE_URL ?? ""
);

const migrationDB = drizzle(client);

async function runDatabaseMigrations() {
  try {
    await migrate(migrationDB, { migrationsFolder: "./drizzle" });
    console.log("Migration completed! ✅");
    process.exit();
  } catch (error) {
    console.error("Migration failed! ❌");
    console.error(error);
  }
}

runDatabaseMigrations();

Got this script command in my package.json

"db:migrate:run": "ts-node migrate.ts"

"db:migrate:run": "ts-node migrate.ts"

Then here's an example .yml for a GitHub Action

name: Run Development Database Migrations

on:
  push:
    branches: ["dev"]

jobs:
  build:
    name: Build and Run Migrations
    timeout-minutes: 15
    runs-on: ubuntu-latest
    env:
      TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }} # I use Turborepo
      TURBO_TEAM: ${{ vars.TURBO_TEAM }} 
      DATABASE_URL: ${{ secrets.DEV_DATABASE_URL }}

    steps:
      - name: :construction_site: Check out code
        uses: actions/checkout@v3
        with:
          fetch-depth: 2

      - uses: pnpm/[email protected]
        with:
          version: 6.32.2

      - name: :construction_site: Setup Node.js environment
        uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: "pnpm"

      - name: :construction_worker: Install dependencies
        run: pnpm install

      - name: :hiking_boot: Run db migrations
        working-directory: "./packages/database"
        run: pnpm db:migrate:run

name: Run Development Database Migrations

on:
  push:
    branches: ["dev"]

jobs:
  build:
    name: Build and Run Migrations
    timeout-minutes: 15
    runs-on: ubuntu-latest
    env:
      TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }} # I use Turborepo
      TURBO_TEAM: ${{ vars.TURBO_TEAM }} 
      DATABASE_URL: ${{ secrets.DEV_DATABASE_URL }}

    steps:
      - name: :construction_site: Check out code
        uses: actions/checkout@v3
        with:
          fetch-depth: 2

      - uses: pnpm/[email protected]
        with:
          version: 6.32.2

      - name: :construction_site: Setup Node.js environment
        uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: "pnpm"

      - name: :construction_worker: Install dependencies
        run: pnpm install

      - name: :hiking_boot: Run db migrations
        working-directory: "./packages/database"
        run: pnpm db:migrate:run

Thanks for the great discussion everyone!

Gaming

Programming

Share migrations in monorepo