Redis re-deploys on new Database services after migration
Hey all, wondering if anyone else has run into this. Since migrating from a Redis database plugin to a database service, we now have Redis rebuilding at odd times (the metrics tab shows the
railway up command, though no one on our team runs that).
Our database didn't used to restart as a plugin and it brought up a case where we had to fix some code being called at runtime.
Is there a predictable reason as to why the docker images rebuild and from what that causes it? (ie I assume it's not a git commit like our python code)38 Replies
Project ID:
1d3ab213-7e4f-45f6-a136-1326d80d0606Project ID: 1d3ab213-7e4f-45f6-a136-1326d80d0606
this is an interesting issue, I haven't seen this before, I've seen people who've accidentally linked to the database service and ran railway up though, but that would also bring down the database
can you provide the service id of the database?
and some screenshots of the activities?
Hmm, we don't tend to run
railway up ourselves. I wondered if some system internally did. Yeah let me get you more details. Thanks for the quick reply!
Service ID: 1f1f51ba-64e8-4350-9b8c-4d0ff6547844
Project ID: 1d3ab213-7e4f-45f6-a136-1326d80d0606 (if it's helpful)These were the logs we were seeing that a user issued a shutdown:
And it was showing as
railway up in the metrics tab like so:
if someone really did run railway up, it would take down the database since a new deployment would be made with whatever code was uploaded via the cli
would you happen to have app sleeping enabled on the database?
Okay, railway up is not a workflow our team uses.
We don't have app sleeping on:
I haven't actually configured anything on the database service since doing the 1-click plugin -> database migration
how many times have you seen a mysterious railway up redeploy?
I migrated Friday late at night. Since then, we've merged code to our API service multiple times, but the Redis database redeploys don't coincide with those times.
The first build was 4 days ago (Fri night), and today it has happened 3 times
super odd, I don't think there's much more I can do here, I'll need to get a team member involved to take a deeper look
Okay would appreciate any help, I sent an email to support as well (just in case this somehow would get merged with a ticket)
Thread has been flagged to Railway team by @Brody.
@matt - redis is randomly re-deploying
(As another datapoint, it just restarted again)
The "Active" is really fucking weird
Do you have a cron syntax set by chance?
No Cron syntax: (looks empty at least, I can put my cursor in the input)
Oh wait active might just be fine and no cron
Very possible we have some crappy code somewhere too, not gonna discount that possiblity 😅
CC @Melissa as I see you just wrapped internally (dunno if you have 5-10m)
I mean tho it's redis on the image
I'll be the first to push it back as "Uhh looks like application issue"
But this IDK how that would be possible....
I wondered if we could be doing some in our code that overwhelmed it or something, but I wasn't seeing hints to that in the Redis logs
It would restart not redeploy if that was the case
Ahh good distinction yes. Since it only starting happening after migrating to the service I thought maybe there'd be something connected.
It's also SO idle
Like, beyond idle
Good instinct and appreciate the flag
(But yea I don't think it's that...)
Yeah we don't hit it super hard. I don't know if context really makes a huge difference but we use it for rate limiting and some simple caching.
fwiw i can reproduce this manually by calling the serviceConnect mutation
Does that track kyle?
Wait serviceConnect as in to GH or Docker?
Or as in shared variable reference
serviceConnect with a docker image
used this https://bookmarklets.up.railway.app/service-redeploy/
We don't call the GraphQL API ourselves (which I assumed that might be related to?..).
I don't know if this is helpful or not but noticed that after I ran the
Reset freeform Quick command I saw the Redis Legacy plugin appear again.
Not sure what connection there was between that old plugin and the newer database service, but maybe I got something stale back.ie my older screenshots in this thread just had Redis, now I see Redis Legacy, Redis, and our API server
(Probably conflating unrelated things, just trying to give plenty of info)
The legacy redis plugin is not deleted which is why it is appearing.
Also, I believe the redeploys happen due to activity in other environments (this is a bug)
I'll try deleting it again, when you mean activity in other environments like new preview environments being opened? I'll see if the times we've created preview environments line up with the times we saw re-deploys...
when you mean activity in other environments like new preview environments being openedyup exactly
Okay yeah that's gotta be it! Seems like it was when we opened up other PRs that created new environments.
Def a bug! Will be fixed soon. But nothing to worry about on your end