Bot protection for Pages doesn't seem to work
I've been on Cloudflare since September 2024 and I enabled the bot protection feature when Cloudflare released it. However, I've noticed that since enabling it, the metrics for my site haven't changed. For example, according to Cloudflare, my site received over 40k unique visitors for the past 30 days (screenshot 1). Meanwhile, Google Search Console only reported 5k clicks (screenshot 2). I know Google Search Console can only display results from Google search, but other search engines can't possibly account for the remaining ~35k clicks. Moreover, this morning I received a report from Google Search Console suggesting that LLMs are actually not being blocked (screenshot 3). Is the bot protection feature stable or is it still experimental? Seems like LLMs are slipping through.
6 Replies
Forgot to add the third screenshot
Free's Bot Fight Mode doesn't block known/verified bots, which includes AI Crawlers like OpenAI and such. There's a seperate option under Security -> Bots -> block AI Bots for that
Yup, the "Block AI Bots" feature is what I was referring to. I've had that enabled for a few months now but it doesn't seem to do anything. Do I need to enable Bot Flight Mode as well for it to be effective?
Additionally even when you enable that, there's still always going to be lots of other verified bots that will be allowed through, like all the search crawlers/engines. And if you're comparing it vs Google's search data, not everyone uses Google + lots of people block trackers
You don't no, you can look under AI Audit in your zone to see requests from them, if any
I see. Hmm, yeah I don't have anything there for the past month. The email I got from Google Search Console seems to suggest someone prompted an LLM and then the LLM used Google to scrape data.
If it's using google/search engine indexes, not a lot you can do about that