6 Replies
They will allowed by default. If you are making firewall rules to block them then you use the
verified bot option in WAF rules to ignore the known good botsAlready found i5
it still blocks the bingbot from crawling "(http.user_agent eq " Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/") or (http.user_agent eq "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) W.X.Y.Z Safari/537.36") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") or (http.user_agent eq "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36") or (http.user_agent eq "Mozilla/5.0 (compatible; Google-InspectionTool/1.0)") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)") or (http.user_agent eq "Googlebot-Image/1.0") or (ip.src eq 20.191.45.212) or (ip.src eq 40.88.21.235) or (ip.src eq 40.76.173.151) or (ip.src eq 40.76.163.7) or (ip.src eq 20.185.79.47) or (ip.src eq 52.142.26.175) or (ip.src eq 20.185.79.15) or (ip.src eq 52.142.24.149) or (ip.src eq 40.76.162.208) or (ip.src eq 40.76.163.23) or (ip.src eq 40.76.162.191) or (ip.src eq 40.76.162.247)" its inside my waf rules if it isnt one of these it skips all the rules
You don't need to do all the user-agent checking. Use
cf.client.bot to get if a verified botWhere
Its that simple
I still get a 400-499 from microsoft while the bot is allowed
Hi, I am getting still same bingbot crawl blocking issue. any help on that?