Database suddenly unreachable
I've been using a Railway Postgres DB for a few weeks. Tonight, while I was busy using the DB with an app I'm building, I suddenly get "failed to connect" messages. When I logged into the Railway dashboard, the DB shows as existing, but it says I have no tables. Under Settings for the DB, trying to restart the DB also fails with "Error processing request" in the UI. Can you help me figure out what happened?
26 Replies
Project ID:
38b29e3b-f318-4d13-a50c-ad2242257e1238b29e3b-f318-4d13-a50c-ad2242257e12
Not good
Working on restoring for you
cool. Appreciate the fast response. it's not a production app (yet) so a little downtime won't kill me. Any idea what happened?
Back
yes it seems to be back to normal.that was fast, thank you
We are performing a massive amount of VM operations to prepare for a new feature rollout (persistent disk) but we are running into vendor rate limits over in GCP. We have done everything we can to protect Team workloads but small DBs are getting a minute or so of downtime.
We did report this in #🚨｜incidents - but apparently the modification of the script is giving us 3m of downtime not 30s so we will adjust. Sorry about that.
Meanwhile, I am keeping an eye out on reports from ya. Atm we have only 5 affected out of millions of workloads- so unfortunately you are one of the unlucky ones.
(If we had more impact, I would declare a maintainence window)
it's all good. thanks for the info. Just FYI I think I had about 15 min of downtime.
Not good at all, lemme credit you.
don't worry about it i'm a tiny app for now.
No workload too small
it sounds like a team plan would improve the situation though right ?
Gave ya $20
like you guys protected those
ok thank you so much 🙏
Yes. But, being honest, its not good optics.
So wanna move to a system where we are more forthright about that system and expectation of service tiers.