Is there a --force option for generating migrations in CI?
Hey guys, thanks for building this awesome tool. Our team loves it. We are using Github Actions and Drizzle to run migrations on a Postgres Docker Container during CI. However, we got stuck at the following:
When running the generate-migrations command, it prompts the user (which does not exist in CI) to select whether to rename a column or create a new one. Since we cannot provider an answer during CI, is there an option to provide a default answer or skip these questions?
Any help is appreciated. Thanks 🙂
12 Replies
Hello 👋
Can I ask why you don't generate the migration during development time? (to include it with the pull request).
@Raphaël M (@rphlmr) ⚡ Thanks for the fast reply! For us it's a matter of consistency + letting people generate the migration files locally adds failure potential to the whole process. Though I totally understand if this is not one of your priorities right now!
I will ask the team if this is something they can/should include 😉
How is generating migrations locally may be potential a failure?
Could you please explain your flow with generating migrations in CI and not locally?
This is our workflow
1. In development
- Team member changes a schema file
2. Opens PR → CI runs in postgres docker container
- Generate temporary migrations to apply the new changes <— The step we are talking about (using
drizzle-kit generate:pg)
- Apply all migrations to the freshly created Postgres instance (using Drizzle migrator)
- Run e2e tests on the temporary instance
- Done
3. PR is merged into main → CD runs
- Generates migrations again
- Applies migrations to real database
- Commits new migration files to the repo
Failure potential lies within the fact that the raw SQL migration files are not as intuitive to review and can simply be forgotten easily. Overall this setup allows us to only change the actual schema and leave the rest to CI/CD, leaving out any possibility for human failure.Does that mean that on local dev, your devs use
pg:push on a local DB to test their code?
It reminds me of some Prisma workflow I saw.How would you suggest drizzle-kit to select between renaming columns or just create them?
Depends on the Drizzle kit implementation behind that but:
--force => what is in the schema (typescript) erases everything?The reason for the
push command, as I understand it, is for devs to test locally without creating the migration.
After they are happy with their changes, they can generate the migration files and test them locally.
I don't know if pushing schema changes without testing locally would be wiseThey test it, they just don't want to generate the migration and offload that in a CI job.
This is common in big companies (with a security team watching everything you push 🤣), a kind of guard to prevent the dev from modifying the previous SQL migration.
If you see some SQL changes in a pull request, it is easy to reject/ask for a revert.
(In a previous job, it was needed to submit SQL modification to a 'security' council)
there is no way to detect if it was a rename or add/delete
that's why it's always should be done manually
I also believe that this approach is a way to reduce errors.
When generating SQL migrations locally and committing them to Git, you can verify that the required SQL file is accurately generated. You can test this script both locally and in development/quality assurance/staging environments before deploying it to production.
There's no need to generate migrations multiple times when you can do it just once locally for your feature.
If you're concerned about someone modifying the SQL script that was applied to the database, it should be checked during the pull request review. Additionally, we store a hash of the SQL files in the database for each migration. While we haven't implemented it yet, we are planning to add hash checks and provide warnings if any inconsistencies are detected.
Drizzle Kit also features the
drizzle-kit check command, which can identify collisions if more than one developer is working on the same codebase. This ensures that migrations were generated in the correct order.I understand. Thanks for taking the time to discuss this 🫶 . We'll adapt our workflow to the intended process. You guys rock!