Helsinki workers down?
Any way to force requests to alternate paths when one worker region is down?
17 Replies
Workers Regions should never be down, are you talking about Workers AI by chance?
no, I have global requests and all of my helsinki requests are failing
I'm not using AI
every other region is not failing
What exactly are you seeing, a specific error or something? Does a really simple worker like https://quick-test.workers.tylerobrien.dev/ work for example?
I don't currently have ssh access to that server, it's a tad complex
Workers run on the same machine as receives the request, it's a pretty uniform setup and every CF Edge Metal is running the workers service. There's no load balancer or anything. Workers being down in a region would be super super critical and be fixed before an alternate path would even be useful
I'll try to log the response headers, one sec
Cloudflare has internal workers like Zaraz, Challenge System, Waiting Room, Image Resizing/Images, Rocket Loader, every Cf API Request, etc that run on every request using the same worker instance, so they're very much invested in Workers themselves
yea, you can also tail the worker if you have access and try to see errors from it https://developers.cloudflare.com/workers/observability/logging/real-time-logs/#view-logs-using-wrangler-tail, either from dashboard (Workers & Pages -> Workers -> Logs or npx wrangler tail).
Something more specific I'd imagine
its not reaching the worker
getting logs right now, one moment
I'm getting a
Forbidden response
it's from a server though, not a client – why is this getting triggered
looks like cf-mitigated: challenge?Check Security -> Events in the cloudflare website to see what service is causing a challenge
well that's all up to your security settings, unless you added a specific exception for server requests identified from someway it's all going to run through the normal waf/security settings/etc. CF's default security settings shouldn't cause that though, the security event will show the specific service causing it
ah its hitting
Bot fight mode but it's certainly not a bot. not sure why that would be triggeredit's an automated request, from a server no? That would be a bot
You can disable it under Security -> Bots
it's my own server hitting my internal api – it's not a known bot hitting any other network
If it was a known bot like Google Crawler for example, those are excluded from Bot Fight Mode
Bot Fight Mode is designed as a high security measure to stop automated requests
Important considerations you need to be aware of before turning on BFM or SBFM BFM and SBFM are high security features intended to quickly help customers under active attack stop as many bots as possible. Due to the high security threshold, false positives do sometimes happen. BFM has limited control. You cannot bypass or skip BFM using the Skip action in WAF custom rules or using Page Rules. BFM will be disabled if there are any IP Access rules present. If you turned on BFM during an attack, and the attack has subsided, we recommend either disabling the feature using IP Access rules to bypass BFM, or looking at Bot Management for Enterprise, which gives you the ability to precisely customize your security threshold and create exception rules as needed. SBFM can be bypassed with IP Access Allow action rules. You can use the Skip action in WAF custom rules to specify where Super Bot Fight Mode should not run.https://developers.cloudflare.com/bots/troubleshooting/ I wouldn't even say that's a false positive though, it's designed to only allow Users/real browsers, not any unverified bots. If you're on Free with Bot Fight Mode, your only option is to turn it off. If you're on Pro or higher with Super Bot Fight Mode, you could create a Custom Rule to bypass it (or turn it off)
that's a bummer. I'm going to move my workers to another domain so it doesnt trigger
that also works lol
thanks for the insight, I really appreciate it
Sure, yea Bot Fight Mode is supposed to be a super strict high security thing, perhaps not highlighted enough in the dashboard itself