Slowness
Hi Team, I'm experiencing slowness in Novu, with DB utilization nearing 100%.
We are using 8GB of CPU for the database and 4GB for the application server, with 2 workers. The application server load is minimal, only 10-30% at most.
Our daily notification count is 14 thousand, distributed throughout the day.
What steps should I take to resolve this issue?
What will be the impact if I increase QUEUE_CHUNK_SIZE?
Thanks
10 Replies
Hi @MP, I'm an AI assistant for Novu. While a team member reviews this question, I'll see if I can find relevant sources for your question.
-----------------------------
While I generate an answer, here are some sources that might be helpful:
* Get the unseen in-app notifications count for subscribers feed
* Introduction
* Introduction to Novu
* Run Novu in local machine
-----------------------------
Answer
It seems that your issue with high database utilization might be related to insufficient resources allocated to your MongoDB instance, particularly if you are experiencing high CPU usage. Based on the information provided, here are a few steps you can consider to address the slowness:
1. Index Optimization: Ensure that your MongoDB instance is properly indexed. Missing indexes can lead to high CPU usage as the database performs full scans over the data. You might want to review your MongoDB logs to identify slow queries and ensure that appropriate indexes are in place.
2. Increase Resources: Given that your database utilization is nearing 100%, increasing the resources allocated to your MongoDB instance might help. This could involve upgrading the instance to a higher specification with more CPU and RAM. 3. Review Worker Configuration: Since you mentioned using 2 workers, consider whether the distribution of tasks to these workers is optimal. Balancing the load more effectively across workers might help reduce the strain on your database. 4. Adjust
2. Increase Resources: Given that your database utilization is nearing 100%, increasing the resources allocated to your MongoDB instance might help. This could involve upgrading the instance to a higher specification with more CPU and RAM. 3. Review Worker Configuration: Since you mentioned using 2 workers, consider whether the distribution of tasks to these workers is optimal. Balancing the load more effectively across workers might help reduce the strain on your database. 4. Adjust
QUEUE_CHUNK_SIZE: Increasing the QUEUE_CHUNK_SIZE can have implications depending on your current setup. This setting determines the number of jobs fetched from the database in one go. Increasing it might reduce the number of queries made to the database, potentially lowering CPU usage if your bottleneck is query frequency. However, it could also increase memory usage on the workers if they fetch more data than they can efficiently process.
5. Monitoring and Logs: Implement more detailed monitoring on your MongoDB instance to get real-time insights into performance bottlenecks. Tools like MongoDB Atlas, if you are using it, provide valuable metrics on performance and can help in identifying issues with slow queries.
6. Database Scaling: If the load on your database continue
s to increase, consider scaling your MongoDB deployment. This could involve scaling vertically (more powerful hardware) or horizontally (adding more nodes).
As for the impact of increasing the QUEUE_CHUNK_SIZE, it could potentially reduce the frequency of database queries, which might help with CPU utilization. However, this should be tested carefully as it could also lead to increased memory consumption on the workers, which might offset some of the benefits if not managed properly.
These suggestions are based on general best practices for managing database performance issues and scaling workers in a distributed system. For a more tailored solution, it might be helpful to consult detailed logs and performance metrics from your environment.
Got it. Tagging @Support for additional help.Additionally, MongoDB queries in jobs are taking a long time.
@Support
@MP, you just advanced to level 2!
Hey @MP, let's see if @Pawan Jain @Dima Grossman, @sokratis can help you here. Can you share a bit about the way you use Novu?
Hi @MP my first suspicion would be missing indexes for the queries.
What indexes do you currently have for the jobs collection?
Hi @Tomer Barnea,
We are using Novu to notify our users Class is booked or Cancelled and also some reminders emails about class. this reminders workflow will run from every 30 minutes to notify users who has future bookings
@Dima Grossman please find the indexes what we have for jobs collection
Yep it defiently seems like you are missing indexes in your database
take a look at job.schema.ts file in @novu/dal package, it has a list of suggested indexes to incorporate depending on your usecases
Thanks @Dima Grossman , Is there any other places where I can add indexes to improve performance?
Shall I increase the QUEUE_CHUNK_SIZE from default?
On all Schemas 🙂
@MP you can try increasing it, to see if this improves performance for you
are you using any observability platform like NewRelic?
This can be helpful to diagnose performance issues
Thanks for the support @Dima Grossman 😇