Database suddenly unreachable
I've been using a Railway Postgres DB for a few weeks. Tonight, while I was busy using the DB with an app I'm building, I suddenly get "failed to connect" messages. When I logged into the Railway dashboard, the DB shows as existing, but it says I have no tables. Under Settings for the DB, trying to restart the DB also fails with "Error processing request" in the UI. Can you help me figure out what happened?
26 Replies
Project ID:
38b29e3b-f318-4d13-a50c-ad2242257e1238b29e3b-f318-4d13-a50c-ad2242257e12
Not good
Working on restoring for you
cool. Appreciate the fast response. it's not a production app (yet) so a little downtime won't kill me. Any idea what happened?
Back
yes it seems to be back to normal.that was fast, thank you
We are performing a massive amount of VM operations to prepare for a new feature rollout (persistent disk) but we are running into vendor rate limits over in GCP. We have done everything we can to protect Team workloads but small DBs are getting a minute or so of downtime.
We did report this in #🚨｜incidents - but apparently the modification of the script is giving us 3m of downtime not 30s so we will adjust. Sorry about that.
Meanwhile, I am keeping an eye out on reports from ya. Atm we have only 5 affected out of millions of workloads- so unfortunately you are one of the unlucky ones.
(If we had more impact, I would declare a maintainence window)
it's all good. thanks for the info. Just FYI I think I had about 15 min of downtime.
Not good at all, lemme credit you.
don't worry about it i'm a tiny app for now.
No workload too small
it sounds like a team plan would improve the situation though right ?
Gave ya $20
like you guys protected those
ok thank you so much 🙏
Yes. But, being honest, its not good optics.
So wanna move to a system where we are more forthright about that system and expectation of service tiers.
yeah it's def possible for someone to play with an individual plan, with intenntion of growing into a fullblown account. so reliability matters
BUT, at the same time, responsiveness matters A LOT too. so i appreciate you guys were here, you fixed it
hope the feature rollout goes well
This is why I do it, for builders like you.
Thanks!
I am personally excited.
We just shipped private networking (type /beta)
and we have regions coming after this
Sounds a little like fly.io ?
Rather these are base primitives that people need to deliver at scale, so a Fly but without upfront config
Don't get me wrong, I love Kurt and team
but they are more below the stack, whereas we try to abstract it all
so our regions impl, is more turnkey rather than you having to copy your config from US-East to EU-West so on so forth
yeah i love how easy it was to set up a DB with you guys
all of app dev should feel that way
if you guys can make it happen that will be a big deal!
for us? more of a when than an if haha
confidence aside, let us know how else we can help
first step to creating the future is believing in it. 🚀 Good luck, will let you know if i need anything else
love this energy
Awesome!
Marking this solved - will keep my eye out for the next fire.