Self-hosted migration scripts in docker
Hi all!
We have self-hosted Novu in docker (docker-compose).
In the description of version 0.18 states that we have to execute the migration script.
npm run migration:primary-provider
What is the correct way to do it in docker-compose instance of Novu.60 Replies
I have the exact same question
Did you manually change the image reference to 0.18 or you are only preparing for the update
up
I'm running with the same issue. I tried to upgrade the api service in docker-compose to 0.18, but the migration doesn't exist in that version (even in the source code that migration doesn't exist in the package.json file in the api folder.
So I checked in the 0.19 version, and there is it. So I tried to upgrade the docker image to 0.19 and run the migration inside the container but it failed because reference another file in the src folder.
So it looks like the migration only works from the source code, and not from the containers.
Lets see what Novu staff says about this.
Thanks
Hi @aegeavaz and sorry for the delayed response here, I would suggest running the migrations from source and not from the containers. (You can clone it locally or in a machine that has access to the mongo instance, and run the migrations)
Thanks. But what about the versions? because that migration is not in the v.0.18.0 tag version but it is described in the release documentation for v.0.18.0. Should we upgrade to version 0.19.0?
You can just clone the next branch, all the migrations file are there historically
Well, it works. Thank you!
@dimagrossman
@novu/[email protected] migration cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/expire-at/expire-at.migration.tsnode:events:491 throw er; // Unhandled 'error' event ^ Error: spawn ts-node ENOENT at ChildProcess._handle.onexit (node:internal/child_process:283:19) at onErrorNT (node:internal/child_process:476:16) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) Emitted 'error' event on ChildProcess instance at: at ChildProcess._handle.onexit (node:internal/child_process:289:12) at onErrorNT (node:internal/child_process:476:16) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { errno: -2, code: 'ENOENT', syscall: 'spawn ts-node', path: 'ts-node', spawnargs: [ '--transpileOnly', './migrations/expire-at/expire-at.migration.ts' ] } Node.js v18.7.0 npm ERR! Lifecycle script
migration failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/[email protected]
npm ERR! at location: /home/technology/novu/apps/api
this is what I get
I am running this against the next branch
any insights to resolve this?@harrisyn based on your logs you try to run the
migration npm script, i'm not sure what version you are upgrading to, but in 0.18.0 the script is: npm run migration:primary-providerHi @dimagrossman
I am running this off the next branch, so you can see the workspace is @novu/[email protected]
might have been assumed, however I think node_modules have to be installed first, this environment had only docker.
What I have tried
1) Ran npm i from the apps/api directory
2) ran npm i from the base novu folder
I still end up now with
npm run migration:primary-provider
@novu/[email protected] migration:primary-provider cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/integration-scheme-update/add-primary-priority-migration.tsError: Cannot find module '/home/technology/novu/node_modules/@novu/shared/dist/cjs/index.js'. Please verify that the package.json has a valid "main" entry at tryPackage (node:internal/modules/cjs/loader:364:19) at Function.Module._findPath (node:internal/modules/cjs/loader:577:18) at Function.Module._resolveFilename (node:internal/modules/cjs/loader:942:27) at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue [as _resolveFilename] (/home/technology/novu/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30) at Function.Module._load (node:internal/modules/cjs/loader:804:27) at Module.require (node:internal/modules/cjs/loader:1022:19) at require (node:internal/modules/cjs/helpers:102:18) at Object.<anonymous> (/home/technology/novu/apps/api/src/config/index.ts:2:1) at Module._compile (node:internal/modules/cjs/loader:1120:14) at Module.m._compile (/home/technology/novu/node_modules/ts-node/src/index.ts:1618:23) { code: 'MODULE_NOT_FOUND', path: '/home/technology/novu/node_modules/@novu/shared/package.json', requestPath: '@Novu/shared' } npm ERR! Lifecycle script
migration:primary-provider failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/[email protected]
npm ERR! at location: /home/technology/novu/apps/api@harrisyn could you please run: pnpm install && npm run build
From the root repository folder
pnpm part is important
Hi @dimagrossman I am still not successful at this.
What I have tried
1) updated the codebase to the latest in next branch.
2) installed pnpm and node (it didn't work with the latest versions, so installed the recommended versions)
3) ran pnpm install && npm run build
- Successfully ran target prebuild for 60 projects (49s)
✖ nx run @Novu/web:build
@novu/[email protected] build /home/technology/novu/apps/web cross-env NODE_OPTIONS=--max_old_space_size=4096 react-app-rewired --max_old_space_size=4096 buildCreating an optimized production build... The build failed because the process exited too early. This probably means the system ran out of memory or someone called
kill -9 on the process.
ELIFECYCLE Command failed with exit code 1.
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
NX Ran target build for 65 projects (38m)✔ 57/59 succeeded [0 read from cache] ✖ 2/59 targets failed, including the following: - nx run @Novu/outlook365:build - nx run @Novu/web:build this process takes waaaaay too long, should I need to increase the VM specifications to run a migration??
I think the issue is with the web projects, which in reality arent required to be built for the migrations
could you try doing: npm run build:api insteald of the global build
And by the way, i'm very sorry for the very complicated flow on this. I would really love to simplify this. Just hadn't a change to get to it yet 😦
oot@Novu:/home/technology/novu# npm run build:api
build:api nx build @Novu/api✖ 1/47 dependent project tasks failed (see below) ✔ 46/47 dependent project tasks succeeded [46 read from cache]
nx run @Novu/outlook365:build
@novu/[email protected] build /home/technology/novu/providers/outlook365 run-p build:*
@novu/[email protected] build:main /home/technology/novu/providers/outlook365 tsc -p tsconfig.json
@novu/[email protected] build:module /home/technology/novu/providers/outlook365 tsc -p tsconfig.module.jsonsrc/lib/outlook365.provider.ts(24,7): error TS2769: No overload matches this call. The last overload gave the following error. Argument of type '{ host: string; port: number; requireTLS: boolean; connectionTImeout: number; auth: { user: string; pass: string; }; tls: { ciphers: string; }; }' is not assignable to parameter of type 'TransportOptions | Transport<any>'. Object literal may only specify known properties, and 'host' does not exist in type 'TransportOptions | Transport<any>'. src/lib/outlook365.provider.ts(24,7): error TS2769: No overload matches this call.
NX Ran target build for project @Novu/api and 48 task(s) they depend on (11s)✖ 1/47 failed ✔ 46/47 succeeded [46 read from cache] @Dima Grossman any alternatives for me? currently stuck here
Distributed Caching and Task Execution for Nx and Lerna
Leverage the power of distribution to bring your Nx and Lerna workspaces to the next level. All nicely integrated into a single unified experience.
@Dima Grossman
@Dima Grossman I am following this thread and wondering how I am going to do this inside a build pipeline with containers in locked down networks.
How do you do it for your production environments?
Cheers.
Hi @todd, in this case you will need to have a virtual machine running in the vpc you are referring to.
I really don't know what to say. We are running this in CI multiple times a day and never have encountered this error. Also tried to replicate this locally with a clean git clone. I have a hunch with something that might be with nodemailer types there, but it still doesn't explain why this happening for you and in none of my attempts to replicate.
Doing a small test, and then will ask you you try fetching again from next
Alright, managed to replicate the error after installing "@types/nodemailer" in the outlook 365 package.
Might be you have some globally hoisted types package present in your environment. issuing a fix right now. Hope it will solve it
@harrisyn pushed a change to next, could you try pulling again and running the build to see if it was resolved?
Will give it a shot, however I tried to debug last night and found
1) outlook365 -> connectionTImeout (typo) should be connectionTimeout - this built successfully with this change
2) errors on line 45 and 48 in packages/application-generic/src/utils/filter.ts, Property 'includes' does not exist on type 'unknown'. -> added a check if it's not null or undefined and it was fine
next error is in passport strategy. I gave up at this point 🤣
@novu/[email protected] build /home/technology/novu/apps/api nest buildERROR in /home/technology/novu/apps/api/src/app/auth/services/passport/github.strategy.ts 19:19-23 [tsl] ERROR in /home/technology/novu/apps/api/src/app/auth/services/passport/github.strategy.ts(19,20) TS2345: Argument of type 'null' is not assignable to parameter of type 'Error'. ts-loader-default_43d7ed5e51b1fe49 webpack 5.88.2 compiled with 1 error in 49811 ms ELIFECYCLE Command failed with exit code 1. @Dima Grossman same spot, synced with next
1) totally correct, also pushed this to next.
2) Do you have some node modules located in your technology folder or home folders?
It feels like your compilation happens in some other ts environment
nope, that environment only had docker installed, I ran the npm run clean command a few times just to be sure and even before each attempt
Argh 😦 @BiswaViraj could you maybe assit here to see if you are able to catch something? Seems like some how the ts compiler for @harrisyn get's access to @types packages that are not available for us during the build. Like in the previous example with nodemailer, or right now.
Just to verify you have only done
?
pnpm i is important and not to use npm directly for each package to properly install dependencies.
Looking into this
Distributed Caching and Task Execution for Nx and Lerna
Leverage the power of distribution to bring your Nx and Lerna workspaces to the next level. All nicely integrated into a single unified experience.
Have you pulled the latest changes from our next branch?
As I can see, it is trying to build
docs which we have removed in the latest version.
Please try to pull the latest changes from our next branch then follow these steps:
1. remove the node_modules folder
2. run npx nx reset
3. run npm run setup:projectSorry, I see this is slightly off topic on this thread—self hosted looks to be probably locally hosted from source. My question is how/when is the migration triggered when running from pre-build images (yes, I have a cluster in a VPC).
I see the migration scripts in the api project. I see that scripts that run them manually (for a local migration).
What I am fundamentally trying to understand is do I need to worry about migrations for my production instance? It looks like I don't but I just can't see in code how the migration scripts are bootstrapped and when. By the api, by a worker? At bootstrap or some interval or other trigger?
Happy to be pointed at the code line and I'll work out the rest. Just struggling! 🙂 Keep up the great work
@todd, you just advanced to level 3!
Currently, we run our migrations manually using the script. As @Dima Grossman mentioned, you'll need to create a VM in your VPC and then just run those scripts
Thanks. I wonder a couple of things. Again off topic in this thread and apologise.
There are three migration commands in that project. So, I wonder what you actually do as well—as in the entire process.
I suppose without this it hard to see that the project can really be run successfully by others in production. It seems to be shame to have a manual steps in deployment pipeline and I will be looking to ensure this is all scripted.
Thanks in advance. (I suspect this all looks obvious to you who do it every day but it is very opaque to me—and migrations are somewhat fundamental)
I agree with you about having automated migrations, maybe we can include it in the docker 🤷🏻♂️ .
For choosing which migration command to run, I will recommend you to follow the version release guides on github as it will have the relevant details about which command to run.
Looking at that documentation, I would like to make a distinction between Upgrades and Migrations. Upgrades definitely "manual" because you need are introducing new features/flags etc. Migrations to me are data migrations WITHIN the existing infrastructure. These are ideally transparent unless the migration infrastructure itself changes which is then an upgrade!
This distinction is all very apparent when the pipeline is fully "automated" (eg say terraform on a build pipeline)
In my own product, I have a migration as a lambda (AWS function with single entry point) that is triggered by the pipeline to run on every deployment (in every environment). So logically the same as first mentioned by Dima and I can do this here.
I just needed to work out the what and was hoping for explicit instructions!
Like I've done with the S3 setup, I will come back when I work out the finer details. I have already had to do this also with MongoDB Atlas.
Thanks for the help
🙂
@todd definitely not off topic, as it was my initial assumption i.e. that migrations can be automated.
It should have been relatively easy if it could be run against a running container and be done - it's how I have done it in my other applications as part of the pipeline.
I can confirm that I am running off the latest, synced earlier today as well, so I saw the fixes I had highlighted previously have been effected into the next branch.
npx nx reset
root@Novu:/home/technology/novu# npx nx reset
node:internal/modules/cjs/loader:1031
throw err;
^
Error: Cannot find module '@nrwl/workspace/presets/npm.json'
Require stack:
- /root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/nx-json.js
- /root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/configuration.js
- /root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/utils/package-manager.js
- /root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/init-local.js
- /root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/nx.js
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1028:15)
at Function.resolve (node:internal/modules/cjs/helpers:125:19)
at readNxJson (/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/nx-json.js:13:48)
at Object.<anonymous> (/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/daemon/client/client.js:319:72)
at Module._compile (node:internal/modules/cjs/loader:1198:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
at Module.load (node:internal/modules/cjs/loader:1076:32)
at Function.Module._load (node:internal/modules/cjs/loader:911:12)
at Module.require (node:internal/modules/cjs/loader:1100:19)
at require (node:internal/modules/cjs/helpers:119:18) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/nx-json.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/configuration.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/utils/package-manager.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/init-local.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/nx.js'
]
}
Your roles on this server do not allow you to use the AI commands.
Reach out to the server owner to know how to get an ai-authorized role or Unlock unlimited uses of this AI-based feature with a personal plan.
got some progress with this. 👍
However 😨
novu# cd apps/api
root@Novu:/home/technology/novu/apps/api# npm run migration:primary-provider
@novu/[email protected] migration:primary-provider cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/integration-scheme-update/add-primary-priority-migration.tsNew Relic for Node.js was unable to bootstrap itself due to an error: Error: New Relic requires that you name this application! Set app_name in your newrelic.js or newrelic.cjs file or set environment variable NEW_RELIC_APP_NAME. Not starting! at createAgent (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/newrelic/index.js:140:11) at initialize (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/newrelic/index.js:81:15) at Object.<anonymous> (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/newrelic/index.js:36:3) at Module._compile (node:internal/modules/cjs/loader:1198:14) at Module._extensions..js (node:internal/modules/cjs/loader:1252:10) at Object.require.extensions.<computed> [as .js] (/home/technology/novu/node_modules/.pnpm/[email protected]_wh2cnrlliuuxb2etxm2m3ttgna/node_modules/ts-node/src/index.ts:1608:43) at Module.load (node:internal/modules/cjs/loader:1076:32) at Function.Module._load (node:internal/modules/cjs/loader:911:12) at Module.require (node:internal/modules/cjs/loader:1100:19) at require (node:internal/modules/cjs/helpers:119:18) Environment: local Platform: Docker Tenant: OS Selected Log Transport pino-pretty Migration for primary and priority fields in the integration entity npm ERR! Lifecycle script
migration:primary-provider failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/[email protected]
npm ERR! at location: /home/technology/novu/apps/api
I suspect the NEW_RELIC error can be ignored, however the migration didn't run successfully, I have added the .env in the same folder i.e. apps/api to ensure it can connect to my mongodb database@harrisyn, you just advanced to level 6!
@BiswaViraj @Dima Grossman
I will apprecaite any further instructions
@harrisyn
From the above error it seems maybe the new_relic caused the error
just to be sure please add this line
NEW_RELIC_ENABLED="false"
to your .env of apps/api/src folder
and them run these commands from the root of the novu project
- remove the node_modules folder
npx nx reset
npm run clean
npm run setup:project
also please verify the node version to be v16.15.1node version is v16.20.2
can you try with v16.15.1
will try that
unfortunately still the same error
I'm re-sending the message again:
Hi @BiswaViraj @Dima Grossman did you guys give up on me?
I am happy to jump on a call if it helps resolve this?
Hello @harrisyn , Sorry about it 🙏🏻
We recently released a new version, can you try using the new version, if it doesnt work let's meet over call
Hi @BiswaViraj, I assume I will still need to run the migration right?
yes
which version are you using currently?
0.17
been trying to run the migrate since then
I understand that is a prerequisite to use 0.18
I just pulled the lastest changes
1) Ran "npx run reset"
2) deleted node_modules
3) ran " npm run setup:project"
Result
nx run @novu/webhook:build
@novu/[email protected] build /home/technology/novu/apps/webhook nest buildnode:internal/errors:478 ErrorCaptureStackTrace(err); ^ Error: ENOSPC: System limit for number of file watchers reached, watch '/home/technology/novu/apps/webhook/src/.env.development' at FSWatcher.<computed> (node:internal/fs/watchers:244:19) at Object.watch (node:fs:2296:34) at createFsWatchInstance (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/lib/nodefs-handler.js:119:15) at setFsWatchListener (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/lib/nodefs-handler.js:166:15) at NodeFsHandler._watchWithNodeFs (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/lib/nodefs-handler.js:331:14) at NodeFsHandler._handleFile (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/lib/nodefs-handler.js:395:23) at NodeFsHandler._addToNodeFs (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/lib/nodefs-handler.js:637:21) at async /home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/index.js:451:21 at async Promise.all (index 0) Emitted 'error' event on FSWatcher instance at: at FSWatcher._handleError (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/index.js:647:10) at NodeFsHandler._addToNodeFs (/home/technology/novu/node_modules/.pnpm/[email protected]/node_modules/chokidar/lib/nodefs-handler.js:645:18) solved the above error by increasing the system limit and setup has run succesfully root@novu:/home/technology/novu# cd apps/api/ root@novu:/home/technology/novu/apps/api# npm run migration:primary-provider
@novu/[email protected] migration:primary-provider cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/integration-scheme-update/add-primary-priority-migration.tsEnvironment: local Platform: Docker Tenant: OS Selected Log Transport pino-pretty Migration for primary and priority fields in the integration entity npm ERR! Lifecycle script
migration:primary-provider failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/[email protected]
npm ERR! at location: /home/technology/novu/apps/api
Migration still fails@harrisyn can you check if the mongoDB connection is correct?
I tried now with an incorrect mongoDB connection and it threw the same error as yours
it is
it is the same as I use to run the docker container and that works just fine
two questions,
1) the .env file containing the mongo_url should be in which folder? i.e api or root?
2) does it matter where the migration command is run from?
1. MONGO_URL .env should be under
/apps/api/src
2. you can run it from /apps/api paththose are both what have been doing so far
When you run npm start does the app work normally in that case?
never tried, it's the same server the docker containers run on, wouldn't there be a conflict?
Ok, yeah it will conflict if it is running in the same containers.
I doubt it is mongo connection
Somehow the api docker isn't able to connect to the DB.
But as it is running in the same containers it should connect.
Also the error doesn't say much,and isn't helpful
@BiswaViraj is there any other means to run this migration, even if manually?
It will be easier to clone the github repo to your local machine, and update the .env MONGO_URL to point it to your prod db, and then run the script locally
Got it, will give that a shot
Are there plans to fix the migrations for Docker? Seems gross that I have to run this from source
Hi @dmgarland adding your feature request to backlog