Bedrock Web Crawler useragent "BedrockBot" blocked
I'm trying to set up a knowledge base on bedrock using our zendesk pages (hosted by Cloudflare). It looks like in the list of allowed bots, Cloudflare allows "AmazonBot". https://radar.cloudflare.com/traffic/verified-bots
However, I just confirmed with the AWS support agent that the user agent used by bedrock is
BedrockBot. Any chance I can get this user agent on the list of allowed/verified bots?2 Replies
The "Add a bot" link on that page or https://dash.cloudflare.com/?to=/:account/configurations/verified-bots lets you.
Bedrock Web Crawler useragent "BedrockBot" blockedYou can always make exceptions in your configuration though, Verified Bots is just a list used by some optional products (Bot Fight Mode), or explicitly by you in the WAF. You don't use it by default, and you can manually whitelist bots in the same way (IP + UA Custom Skip Rule)
Gotcha - we're using Zendesk which is hosted by Cloudflare, so I don't have a list to their configurations. However, I'll reach out and pray that they change that