Self-hosted isntance of mongodb using 200% of CPU when sending ~1500 notifications through topic
Hello! My team decided to self-host Novu. We are using a t3.medium Amazon EC2 instance and we have like 1500 users provisioned to Novu and there is one workflow that is used to send notifications to all of those subscribers. When we trigger the event to notify all 1500 subscribers, the usage of our mongodb instance increases to 200% and returns to normal until the workflow finishes sending all 1500 notifications.
Do you have any leads on what could be happening? I don't know if we could have missed some configuration of the server or of mongodb, maybe setting the usage limits or something like that.
I'll leave some screenshots with stats from our server if that helps you get a better view of the problem:
1. A graph of the CPU usage right around the time the massive sending took place.
2. The usage of the mongodb process on our server.
3. The mongostat command
6 Replies
My assumption is that you are missing some indexs.
When you set up your system what docs did you follow?
did you set
in your env file
@Zac Clifton where's that one documented?
It isn't in the .env file in the deployment folder
https://github.com/novuhq/novu/blob/d4533be0e0184a1c7764db0df68657dd5fdfd730/docker/.env.example
Can I note that that is the default setting.
https://github.com/novuhq/novu/blob/d4533be0e0184a1c7764db0df68657dd5fdfd730/libs/dal/src/dal.service.ts#L11
@Bocorbojon hey, there is another thread that covered this issue off too if you can find it. Sorry don't have it at hand
I only know about this due to me hitting something similar in staging environment
https://github.com/novuhq/novu/blob/d4533be0e0184a1c7764db0df68657dd5fdfd730/libs/dal/src/dal.service.ts#L11
GitHub
novu/libs/dal/src/dal.service.ts at d4533be0e0184a1c7764db0df68657d...
🔥 The open-source notification infrastructure with fully functional embedded notification center 🚀🚀🚀 - novuhq/novu
In a production use-case at our scale, we can not do this automatically as an index can severely impact response time.
So the recommendation is to add AUTO_CREATE_INDEXES=true in the .env file, or to not add it?
If you are doing over 250 notifications a second it is recommended you apply the index by hand, else turn it to true in your env so all of the indexes are applied at once.