Slow down in performance / Redis / self host
Hey, we've got a staging and production setup of Novu running on AWS ECS. When we've tested and tweaked things in the staging environment, we simply push the docker image to production and get it up and running there.
But, here's the thing: lately, we've noticed that the production instance (which isn't heavily used right now) tends to slow down over time. Certain endpoints like
feed and markAs take up to 40 seconds to complete, even for users with less than ten notifications.
However, we stumbled upon something interesting today. We found that if we restart our Redis instance (which we don't normally do during deployments), Novu instantly becomes more responsive.
I am unsure if the slow down is something that happens over time on its own, or if it is related to not restarting Redis when a new deployment takes
Anyway, just wanted to share this observation with you. Let me know if you have any thoughts or suggestions.4 Replies
@zacclifton Can you please help here
That sounds like what is happening is that you don't have enough workers to process all of the job events from redis. If you are only using 1 redis instance then restarting your redis instance would clear all scheduled jobs and data stored there from the system which would decrease the amount of jobs needed to process on the workers.
If you find something contrary to this please let me know.
Thanks for the feedback, I will look into it. I am unsure why
- there would be so many tasks on the Redis queue when there is very little activity. (We have one workflow, no cronjobs / scheduled jobs or digests)
- a rest endpoint (
markAs) would be slow due to worker tasks, other rest endpoints appear to be fine (seen)
We will look into it more and update here with our findings. We are in the process of moving from v0.12 to v.016, hopefully this will help.
Is there a simple way to see the outstanding tasks at a given point in time?Unfortunately, not without paying for the taskforce.sh monitoring tools