Is there a migration path from v0.x to v2.x self hosted?
I'm currently running self hosted 0.23 in production and have ~30 workflows.
Should I in theory be able to migrate to 2.x? Seems like lots has changed between them but I haven't seen anything explicitly say you can't migrate, but also can't seem to find docs advising what that migration looks like.
Is there breaking changes in the data model between them?
12 Replies
How did you get on with the migration? Tips, tricks or notes?
Hi, this is off topic but is it possible for your to share amount of triggers/notifications you send daily/monthly?
And are you seeing high request duration for queries on
jobs collection?
(I've been facing performance issues from a while so wanted to see if others are having issues or not.)
Thank you 🙏Hey @Dima Grossman @Pawan Jain ,
I’m really struggling to cut through the noise around how Novu handles migrations — and I know I’m not alone. I’d genuinely love to help improve the documentation, but I’ve been trying to figure this out for over a year (on and off), and it’s still unclear.
If this is documented somewhere and I’ve just missed it, I sincerely apologise and would be grateful for a link.
From what I can tell, there seem to be internal processes for how and when migrations are run. That makes sense for your hosted infrastructure, but for those of us self-hosting, it’s a real gap. @Prosper once mentioned that it “just works”, but I haven’t found any evidence of automatic migrations in the code.
If migrations are manual — that’s totally fine — but then:
• Where is the source of truth for how and when to run them?
• Is there a defined order?
• Is there any CLI command or hook to help trigger them safely? (seems like we need a custom deply)
If my understanding is correct and there’s no public guide yet, I’d be happy to document what I find and contribute that back to help others.
Not knowing how this works makes it harder than it should be to maintain both a self-hosted Novu instance and an SDK (in my case, .NET). I’d really appreciate any guidance you can offer.
Thanks so much,
Todd
For anyone interested, I have started to compile a set of notes, including knowing which migrations to run.
@todd, you just advanced to level 6!
Hi @todd, you are 100% correct, there is currently no automated migration in place that is available. And migrations are ran during deployments. We are trying to keep updates without any migrations needed, unless it's completly required and than it will be noted within the release notes. With v2 API, is still compatible with older workflows and notification center.
How-ever, we are currently introducing the new dashboard to be available with the self-hosted edition, which will open a new workflow management paradigm, which works with the v2 endpoints. Once we have this available we will provide a guide of migration v0 workflows to v2 workflows, but this will not be required, as the older workflows will still continue to work.
There is a documenation page available here: https://docs.novu.co/community/self-hosting-novu/data-migrations
I've just noticed that it was not properly migrated to the new docs provider so it was only avialable on the v0.x docs, feel free to issue a PR against this page with any additions 🙏
So just to wrap it, once the new dashboard UI will be avilable on the v2 release, we will provide a proper migration guide for your workflows.
Data Migrations | Novu Documentation
Learn how to update your database data through migrations.
Thanks for the answer. The link to the migrations page was key as the current link in the release notes is broken.
@Dima Grossman Can I also clarify that:
* there is no practical way to run the migrations from a docker version of the api image. It must be from source.
* for all your environments (from local development all the way through to productioin), do you have migrations in any build pipelines? Or does everyone and each environment migrated from a developer machine or dedicated machine?
ps I would love this to be wrong but I have tried and the migrations are looking back through to the developer setup.
@todd internally we have a custom deployment service that runs migrations on AWS and is triggered by our team. As mentioned, by policy any changes are backward compatible and non schema migrations is needed for any deployment. So any migration can be run after the service was already deployed unless specified otherwise. Those migrations are usually for cleanup purposes, and schema alignment.
And yes, there is no practical way to run those from within the docker container at the moment
@Dima Grossman thanks so much. I will get to making a submission for documentation as I have been upgrading all my environments and working out the glitches/gotchas/practicalities for self-hosted:
* migrations prior to 0.19 seem not to exit properly
* some migrations require redis—do you expect that this is real side effect (ie it must connect with the actual redis?)—if so, then I will need to improve the script and include tunnel instructions
* for interest, is there a reason you have never included a migrations runner to manage these (noted in documentation)
* general comment, migrations in my experience are also projects/images and also I wonder why this hasn't been separated out (I suspect oversight/time—document databases still need migrations as you can see :-))
Finall comment, in my documentation I have reconciled the documentation against what can be seen in the git respository for long-term maintenance. The current once is probably correct, but hard to reconcile. Just a heads up of the why I did it as I have.
Thanks. (guide included now for back reference)
For those tracking: https://github.com/novuhq/docs/pull/825
GitHub
fix: migration steps for self-hosting by toddb · Pull Request #825...
Update the instructions based on self-hosting that is tested and starts from no source code. Includes a shell script for managing the process and also commands for helping maintain the table of mig...
Thanks @todd !!! @Pawan Jain let's add that to the upcoming work about migration
Do you know anything here around the migrations?! https://discord.com/channels/895029566685462578/1181954704046034994/1364840582060576870