Crawler Detection in Cloudflare Worker
Can we revisit this? Any way to detect IF crawler using cloudflare worker on PRO plan?
24 Replies
Thanks for any help 🙂
Transform Rule -> Modify Request Header -> this -> Read
X-Verified-Bot header in your WorkerThis is done
Can you please assist with Bot Level Code?
Simple as this?
const bot = req.headers.get('X-Verified-Bot');
if bot = ('Bot1|Bot2|Bot3|Bot4')
bot is just true or falseREVISED:
const bot = req.headers.get('X-Verified-Bot');
if bot == true {
you win
}
@kiannh
cf.client.bot isn't pro?Anyone can have it
interesting, any other headers you know of that are helpful to pass in like that? Currently I pass in the timestamp, gonna add
X-Verified-Bot tooAvailable fields and functions for HTTP request header modification...
The available fields when setting an HTTP request header value using an expression are the following:
See if any of those sound interesting for your use-case
Of course,
cf.bot_management.* is Enterprisehang on, why can't we use botManagement?
erm I mean
req.cf.botManagement.verifiedBotbecause that's an Enterprise add-on
if you mean it's available to you right now when you're not on an Enterprise plan, don't rely on that staying
how's that look?
ah many of those are ent only
this is what I ended up with
Wait, guys, did I get this right?
const bot = req.headers.get('X-Verified-Bot');
if bot == true {
Can someone confirm please, not too familiar with workers?
Seems fine to me
Only good bots will be
true - so stuff like Googlebot, Bingbot, etc
Any random scraper will still be falseThat should stop them 😄
Google banned our Shopping Account because they cant see pricing, which we hide for most visitors. Now, the goal, is to show them pricing so they can leave us alone 😄
doesn't google also ban if they figure out you're offering them different prices than everyone else?
Yea, they're sensitive. Thats not the case tho
To elaborate, we hide pricing for normal users. When they are sent the /pricing/ link a cookie is inserted that enables pricing on main site. Just want to hide pricing from the normal user is all.
This pissed google off due to "Checkout Experience Broken"
So hoping above code will enable pricing/set pricing for google crawler and they can chill.
We aren't lying, just giving them access