R
Railway13mo ago
Connor

Intermittent Network Errors

I am receiving intermittent errors from my service today. There have been no configuration changes to deployment or restarts in the ~2 last weeks. This is just happening today. The service is load balanced with 2 replicas, and there are no errors in the logs for them.
Method: GET
Status: 503

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: immediate connect error: Cannot assign requested address
Method: GET
Status: 503

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: immediate connect error: Cannot assign requested address
10 Replies
Percy
Percy13mo ago
Project ID: 6d146cc0-b14a-41e6-80f6-6d0a0baf99d7
Connor
Connor13mo ago
There appears to be AWS outages in us-east-1; does Railway use AWS?
Brody
Brody13mo ago
they use gcp
angelo
angelo13mo ago
Do you use any AWS services? like S3?
Connor
Connor13mo ago
No, I do not use any AWS services.
angelo
angelo13mo ago
Strange, going to flag to the team
Connor
Connor13mo ago
Thank you!
JustJake
JustJake13mo ago
Hey Connor! We actually identified a DDOS to our platform that about lines up with your issue. Very sorry about that. It was resolved promptly and we've added a card to discuss on Monday how to get ahead of these to prevent stuff like this in the future Please let us know if this pops up again, and we'll follow up as we roll out changes here
Connor
Connor13mo ago
Thank you for the update and promptness at figuring it out! I will keep an eye out if it happens again. Turns out it only happened for 11 requests. Super excited for everything yall are rolling out. Just switched over to multiple replicas instead of Cloudflare Load Balancing and its been going great!
JustJake
JustJake13mo ago
Great to hear! Ya that looks about exactly where we saw the DDOS blip ;_;. Sorry about that, we'll make it better!