All services down in Singapore
Hi team, I've messages my business class team but just for more visibility, our databases are down across all our projects by the looks of things. They've crashed and don't appear to be up and running yet (it's been 17 mins)
Project 3e218ec0-7c31-4a5e-ba63-d0f7439aa07d
Service 8ec6131b-e177-40ec-824e-9fa9f3960add
8 Replies
Project ID:
3e218ec0-7c31-4a5e-ba63-d0f7439aa07d,8ec6131b-e177-40ec-824e-9fa9f3960addOne project is live as of right now, checking the others
A single host in asian-southeast1 encountered a hardware failure. W...
A single host in asian-southeast1 encountered a hardware failure. We've failed over to a new host. - Incident details - Railway Status
please go ahead and redeploy your crashed postgres databases
Hey Brody. Thanks. They redeployed on their own, but the services that relies on them have had to be redeploying to get everything working. Restarting didn't seem to work
I'm bringing them up one by one
let me know if you run into issues during that process
Thanks Brody. All looking good now. This can be closed. Thanks team for the fairly quick resolution – 18m 28s downtime in the end at a very fortunate time of the day for our business
sorry for the inconvenience, hopefully the team can implement a fix so that this particular error doesn't happen again