Redis keeps crashing
Hey everyone,
We migrate our infrastructure on Railway.
The problem is that Redis is crashing and we don't know how to solve, there is no crash logs and we just see Memory spikes.
We also saw a warning and want to add 'vm.overcommit_memory = 1' to /etc/sysctl.conf but don't know how to do it in the interface of Railway.
How can we avoid these crashes ?
40 Replies
Project ID:
66a8764a-14a2-4e1a-980a-79971c3d9cfb66a8764a-14a2-4e1a-980a-79971c3d9cfb
looks like redis is trying to use more than 32 gb of ram
can we avoid the crash when he try it?
since you cant configure anything in redis.conf, the answer would be an unfortunate no. maybe we can pull the team in on this and see what can be done
@Angelo - redis crashing with out of memory on the teams plan
should i try with my own dockerfile containing a redis ?
You can, but the crash is coming from the OOM
no docker volumes yet, so thats not feasible
what should i do?
use an other service for redis?
lets see what the team can do for you here
I will need more info on what you are using Redis for
and in the meantime, bump the container limit beyond 32GB
but I don't have much info from you as much as I'd like to help
we are storing some data in cache and also use it for pub/sub client
the data are mainly json object
and some railway app are connected to redis and publish these json (around 50 json per sec) and some data consume it as subscriber
Is this answer your questions?
We had kubernetes before and the infrastructure with redis was working pretty well
Yep, forgive me as I am coordinating other issues, you can either deploy a Dockerfile (we are shipping volumes this month) with your own tuning or I can raise the limit to help you deal with the OOM issue
im having trouble deploying a dockerfile
i dont know how to expose the docker
and connect from other railway apps
i can connnect myself from localhost
but when i add a domain
and try to connect on it
its not finding anything
you are trying to deploy your own redis?
yes
im really not sure why angelo told you to try to deploy your own redis instance, it cant work, for 2 reasons:
1. theres no persistent storage yet, data would be lost on redeploys
2. you wouldn't even be able to access the redis instance, railway's proxy will only proxy the http protocol, redis uses RESP
that’s for when we beta launch persistent volumes, which is super soon 😛
I’ll bump the memory limit for you to 64G
Is this over Redis pub/sub?
Alright done, your Redis now has 64gb mem limit.
vm.overcommit_memory wouldn't help in this case, the issue is you're hitting the upper bound of the limit. Do you happen to know how much memory this was consuming in your previous kube deployment? Curious if there's some tuning that would help here; looks like you just have lots of data.
And anyway, this is Redis so lack of persistent vol is totally ok (unless you're relying on persistence with AOF). You can just deploy a custom instance with your tuning if you like, but you'll just run into the same issue here.Ok didn' understood it well sorry
It's keep crashing every 5 min ..
Yes redis pub/sub
what does the memory graph look like now?
In the previous one, have 1 node pool with 7 nodes of 2.5 RAM and 2 CPU each
I dont rely on persistance so its ok
oh Angelo was probably guessing you would communicate to your own redis instance through the internal networking
then yeah, for sure go ahead and spin up your own customised redis service
i have some trouble to exposing it from railway
to public
you don't need to expose it publicly, that won't work anyways
how can i connect others services to the redis?
did you not hear about internal networking??
we have some other cloud providers
well railway has it too now
https://discord.com/channels/713503345364697088/921233523719946260/1119371998452264990
this allows you to run your own redis instance
makes sense
do /beta for access
yeah but private networking will only makes the redis accessible from railway apps
i would like to expose it so i can access also from some aws apps
ah gotcha, exposing services that run something other than http isn't possible right now
is there maybe possibly some in app tuning you could do with your code?
to limit reqs per sec on redis?
something like that, or manually clearing the cache on redis?
the problem will be the same, the redis will grow and instead of taking 5 min to crash it will take 20 min to crash
and we dont store pub/sub in the cache
it should quickly wipe
the cache used is less than 500 mb
then what's causing memory to spike above 64gb? that's frankly an insain amount imo
^ needs to be answered first, because the issue is you're running into memory limits. Somewhere, your application is pushing a gigantic amount to Redis (and then flushing it by the looks of your metrics).
We're happy to raise that limit for you, but you'll just run into the same issues again 😕