Critical Cloudflare 503's on Express Backend
Two quick questions on outages I've using Cloudflare + Railway
1. Does railway do any rate-limiting? If so, can this be disabled?
2. Is there a more lengthy post about issues when enabling cloudflare for backend services? (beyond what is mentioned here https://docs.railway.app/deploy/exposing-your-app)
What we're seeing is that immediately after enabling the Cloudflare orange cloud, the express backend chokes on 100% of requests returning 503 errors. I've had many back-and-forths with the Cloudflare team and the current guess is that the origin server (Railway) is throwing these 503s.
36 Replies
Does railway do any rate-limiting? If so, can this be disabled?no, it doesn't do any ratelimiting
Unknown User•3y ago
Message Not Public
Sign In & Join Server To View
Did you see the Cloudflare section at the bottom?
Where you need to have it set to full not flexible?
We don’t do rate limiting
I have it set to full not flexible
api.earni.fi is where all these issues are happening
Project: 46b499ee-c469-4cd0-8d59-e80657e4db3c
Service: 90a80fa7-879c-4463-8675-e3a1aeeafc8c
Unknown User•3y ago
Message Not Public
Sign In & Join Server To View
Short-term outage, apologies. Upstream API went down that my whole app relies on. Should be fixed soon
Unknown User•3y ago
Message Not Public
Sign In & Join Server To View
Fixed! Ready for your thoughts 🙏
https://api.earni.fi/v1/health
Unknown User•3y ago
Message Not Public
Sign In & Join Server To View
No, because I've disabled cloudflare
The moment I enable cloudflare I get all the 503's included in the chrome inspector screenshot above
Can you put cf infront of api2?
So we can see
I cannot. If I enable cloudflare at all (grey cloud) it will still 503 all of "api.*"
Right but do it on api2?
Or add another route that we can check on?
Without repro it’s impossible to check
Here is Cloudflare support's most-recent message
I've been going back-and-forth with them nearly 2-months on this issue
We have a boatload of people who use orange cloud on Railway
So this is quite odd
It seems to be a config issue somewhere
Because this is very possible
Config within my cloudflare configuration?
If cloudflare were doing the 503's, it would be a cloudflare error though
Likely, since the Railway setup is pretty straightforward
But I need to narrow it down
Can you add an api3 that just points to api? And turn on orange cloud for it?
This first began with someone DOSing my systems two months ago. Thousands of requests per minute
I cannot, because currently cloudflare is "paused" at the global level for the entire domain. All routes are resolved with only the cloudflare nameservers. If I enable any cloudflare DNS, it will 503 the primary "api.*"
Everything is set to grey cloud in the above example ^
But you can do it on a per URL level from the DNS section right?
Like what does your DNS page look like
Ya don’t do that
Go to DNS and toggle it on a route
Turn on orange cloude?
On just that one route
If cloudflare is disabled it won't send traffic anyways?
I'll enable api2 now
If you need api and api2
Add api3
I do not need api2, that was the point of making it
Enabled.
Perf
Kk let’s see
Will check back in an hour, going into a meeting 🙏
And you're verifying that is indeed going through Cloudflare? I don't know how
api2.* could be using the cloudflare orange cloud if the global "advanced actions" of disabling cloudflare is still set to Disabled
Hey folks, api2 does not have cloudflare working still. Since my global config has cloudflare disabledYa saw that quite odd :/
Do you have another domain you could point there and enable it?
Hard to test what’s up otherwise :S