Railway•2y ago

Intermittent Network Errors

I am receiving intermittent errors from my service today. There have been no configuration changes to deployment or restarts in the ~2 last weeks. This is just happening today. The service is load balanced with 2 replicas, and there are no errors in the logs for them.

Method: GET
Status: 503

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: immediate connect error: Cannot assign requested address

Method: GET
Status: 503

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: immediate connect error: Cannot assign requested address

10 Replies

Percy•2y ago

Project ID: 6d146cc0-b14a-41e6-80f6-6d0a0baf99d7

ConnorOP•2y ago

There appears to be AWS outages in us-east-1; does Railway use AWS?

Brody•2y ago

they use gcp

angelo•2y ago

Do you use any AWS services? like S3?

ConnorOP•2y ago

No, I do not use any AWS services.

angelo•2y ago

Strange, going to flag to the team

ConnorOP•2y ago

Thank you!

JustJake•2y ago

Hey Connor! We actually identified a DDOS to our platform that about lines up with your issue. Very sorry about that. It was resolved promptly and we've added a card to discuss on Monday how to get ahead of these to prevent stuff like this in the future Please let us know if this pops up again, and we'll follow up as we roll out changes here

ConnorOP•2y ago

Thank you for the update and promptness at figuring it out! I will keep an eye out if it happens again. Turns out it only happened for 11 requests. Super excited for everything yall are rolling out. Just switched over to multiple replicas instead of Cloudflare Load Balancing and its been going great!

JustJake•2y ago

Great to hear! Ya that looks about exactly where we saw the DDOS blip ;_;. Sorry about that, we'll make it better!

Gaming

Programming

Intermittent Network Errors