Redis Socket Timeout
Hey guys!
Been loving railway. Finalizing our production deployment for the release of our product. We have heavy reliance on Redis, and i have been noticing that our Redis instance seems to regularly kick off connected clients (ie our other services). Is there something I can set up for add to our redis instance to fix this?
Thanks!
16 Replies
Project ID:
6fb8606f-116c-4e15-b653-24c48bdad9dd6fb8606f-116c-4e15-b653-24c48bdad9dd
Any possible help on this? I tried migrating from Redis to IORedis and thought this was gone, but am still seeing it come up.
I have a Read/Write timeout set to 3 seconds, that seems to work fine for my apps. I also connect to redis over private url.
How did you configure your timeout?
Any idea on this? The same code has no problem running against Redis cloud on heroku and other services. But for some reason my connections get kicked every 24-48 hours. And it happens at the same time in both my dev and prod environments.
the timeout would be configured through the redis client you are using
@Brody The error is actually not a socket timeout look at the logs. It is a 'connecton reset".
please have the redis client do connection retries
this is likely due to your app trying to write/read from a closed connection, either add retries or make sure to fully close idle connections
I have a progressive backoff strategy implemented which tries to reconnect at every increasing periods of time. It looks like it manages to reconnect, and then gets booted again.
make sure to fully close idle connections
or you could try doing a ping every few minutes to keep the connection alive
Every agent (AI Agent) we have in our system pings its health every few seconds which is picked up by an agent manage to ensure all enabled agents are healthy, so it is a very active redis instance, in theory.
Looking through this thread right now:
https://github.com/redis/node-redis/issues/2032
GitHub
SocketClosedUnexpectedlyError: Socket closed unexpectedly · Issue #...
Good day! The socket connection closes unexpectedly and doesn't reconnect, my older version of node-redis(2.8.0) was reconnecting fine and still does. After reconnect it doesn't print ready...
are you using v4?
either way, these comments are for issues with redis database hosted in various places, meaning this isnt an isolated to railway issue
It seems to often be related to issues in the setup of the redis instance itself, at least in some cases. Just happened again.
What is weird is that it happens at the exact time time in two separate environments on two different redis instances
I havent had this issue on other managed redis services, so I may just swap out to use something like Upstash for now until I can determine what is going on. At this point there is not much left I can modify in my code to prevent it or handle it.
the github issues page you linked has some fixes
I'd also like to note that the redis on railway is not managed
Thanks!