Bypassing bot protection
I'm writing a CLI tool that crawls websites we make to check for broken links and other items. Naturally, Cloudflare sees this traffic, considers it a bot and blocks it after a few requests (usually).
Can I bypass this protection with a header and API key or something along those lines? I'd prefer not having to allow an IP because I want the tooling to work wherever I am.
6 Replies
Which service is blocking? Bot Fight Mode (free), Super Bot Fight Mode (Pro or higher), or something else? You can check under Security -> Events
That's a really good question. I had just tried it on a new site and it's not getting blocked right now. A while back when I was first developing this tool I was getting blocked.
Should I keep playing around with it and when I run into this issue, visit the area you mentioned and go from there (reading docs etc)?
Maybe the sites I was testing it on a while back had bot protection enabled and these don't? I assumed it was something that was just enabled by default for any Cloudflare proxied site
If it's not your site you can't do much. If it is your site/you have control over it, depending on whats blocking it, you may be able to do something.
Free has Bot Fight Mode, a super strict bot blocking solution which you have no control over (on/off). Pro or higher has Super Bot Fight Mode which would allow you to make a Custom Rule to bypass it (on a per-site setup) based on User Agent, or some special header you send, etc
Yeah, they're our sites. Glad to know which products affects it so I can look into it more as I come across it
the default CF config is aimed to be super friendly and not block anything unless it's sure it's malicious traffic/ddos/etc, so that people can onboard without having to adjust things in most cases
It could also be a Custom Rule or something else you setup yourself, but yea those two Bot Fighting products are the most likely to cause it.
Thank you!!