Railway Incident Megathread
During a Railway Incident, post any issues you have here and include as much detail as possible. Please do not create additional threads.
36 Replies
Project ID:
N/A31 minutes since status was posted. Can we get an update please?
PS We're not on the new Edge Proxy and all our systems are down in Singapore. Perhaps this is the "TCP Proxy" purple bar? Just that the wording doesn't align
We're online as of a few seconds ago
The only info I have is that the team is aware of and is working on the issue. Please stay tuned to #🚨｜incidents for more info
If you're online, sounds like they're making progress
ac66a99b-0059-4a75-b6b3-dc683459b0e4
Hugops to the team!
One issue I have is a Redis service that starts but can't connect to it
no errors on the logs, I even get
That would be related to the incident. If you're still having issues after the incident is resolved, open a #✋｜help thread
Yeah I think it is
New reply sent from Help Station thread:
We're seeing some recovery as we're rolling out the fix, still not 100% back.You're seeing this because this thread has been automatically linked to the Help Station thread.
glad to hear it, thanks rems
New reply sent from Help Station thread:
During the incident I saw that the network consumption skyrocketed, causing the charge to skyrocket as well, and the limit did not work.You're seeing this because this thread has been automatically linked to the Help Station thread.
That'll be something the team works out after the incident is resolved, but thanks for letting us know
My services all started working again, would love a post mortem on this since it downed my entire prod systme
my postgresql still down
all services are up now, except for my Mongo db
New reply sent from Help Station thread:
my mongodb still downYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
same here. mongodb still downYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
( I like the service that Railway offers, but it is still a very immature company. There have been many errors lately. I'm seriously considering migrating to DigitalOcean ) 🫤You're seeing this because this thread has been automatically linked to the Help Station thread.
same for me, Redis still down
New reply sent from Help Station thread:
mongodb appears to be working nowYou're seeing this because this thread has been automatically linked to the Help Station thread.
Network usage skyrocketed and the usage limit did not work during the incident
New reply sent from Help Station thread:
same here. Thanks team! @railwayYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
mysql service is still not accessible for me since the incident despite showing as onlineYou're seeing this because this thread has been automatically linked to the Help Station thread.
The team is aware, you will be compensated
The incident is still active, your services being offline is expected
New reply sent from Help Station thread:
One of my services was available for 10 minutes and now not anymoreYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
looks like it's working now. Hey Railway team, it may not be as simple as this but y'all should roll back to the legacy edge proxy that was working while you work on the fix. Once the fix is up deploy it. Downtime of an hour is critically bad and makes me consider going somewhere elseYou're seeing this because this thread has been automatically linked to the Help Station thread.
Redis is back for me, thanks team !
New reply sent from Help Station thread:
network usage for me went up as wellYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
yeap, 1 hour downtime is bad, I am glad its all up now (at least for me)You're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
i guess it's only mb lolYou're seeing this because this thread has been automatically linked to the Help Station thread.
@Adam Will cost increases be taken into account on a case-by-case basis or how will the process be carried out?
that’s up to the team, I have no visibility into how they resolve cost issues
help me
#🚨｜incidents
I accidentally deleted my old proxy and is it possible to see which one was old?
This mysql wasn't working for hours long before everyone went down
if your problem happened before the incident, please open a separate #✋｜help thread. Provide as much detail as possible
New reply sent from Help Station thread:
Are all the servers up again? because our server is still downYou're seeing this because this thread has been automatically linked to the Help Station thread.
the team has reported that the proxies are nearly all back up. If you are still having issues, please post in here and give more details
#🚨｜incidents
New reply sent from Help Station thread:
Every region is now back up. We will be writing an incident report and sharing it by end of week (aiming tomorrow)Apps using the tcp proxy for database connections might need a restart to kick back up and reconnect to the database(s). We are expecting some elevated latency as traffic is going back up, but there shouldn't be any major outage left.If you're still running into issues now, I would invite you to create a new thread with the issue so we can take a quicker look at resolution. I will also be going through this threadYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
What extra information do you need? because our stage server is back up, but production isnt. Or does it just need a few more minutes?You're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
How can we avoid this issues i future? I really love railway but it was never stable :/You're seeing this because this thread has been automatically linked to the Help Station thread.