Redis connection is getting stale on idle
We self hosted Novu on Azure and we are currently on version 20. We are using Azure Redis and noticed that if the worker app is idle, the redis connection is getting stale and have to restart the worker to process the messages. Is there a setting to keep the redis connection alive even though there is no traffic?
11 Replies
After how long do you notice it going stale?
Also it looks like you could set the REDIS_KEEP_ALIVE env variable to set this to a lower number
https://github.com/novuhq/novu/blob/e888590b9204c6ff3b8319c184dfcbc9003ddf80/packages/application-generic/src/services/in-memory-provider/providers/redis-provider.ts#L22
GitHub
novu/packages/application-generic/src/services/in-memory-provider/p...
🔥 The open-source notification infrastructure with fully functional embedded notification center 🚀🚀🚀 - novuhq/novu
assuming you are using a single redis instance
Otherwise if your running two redis clusters, 1 for cache and one for queue then you would set the REDIS_CLUSTER_KEEP_ALIVE env variable
@Zac Clifton I think Azure Redis has a timeout of 10 minutes by default.. that's when it is going stale. The Keep Alive variable is set to 30000ms by default which is 30 sec but still the connection is going stale. Does Novu have an ability to ping the Redis for every few minutes to keep it alive?
We do not have a cron job to do this if that is what you are asking, we are currently relying on the keepalive feature at this time
does keepalive make sure the connection is not getting closed?
@Ujwal, you just advanced to level 4!
do we need to explicitly set it in the env vars since the default is 30 secs anyway?
I would recommended tring this, we have not had this problem on our side yet as we deploy in aws but I am happy to continueing helping as much as I can on the side.
sure I will try it and let you know, thanks Zac!
My pleasure