worker to allow web scrape by friendly bot?
I'm using an AI app called Mendable to scrape my own website for training. CF is blocking their scraping bot when it tries to procedurally visit every page based on the sitemap. Mendable claims it would be "impossible" to know the IP range for their scraper, so I don't think I can whitelist the bot.
Is there a way to write a simple worker that would allow this "friendly bot" to scrape my website?
1 Reply
I just googled the problem and it looks like you can add a whitelist rule https://community.cloudflare.com/t/how-to-whitelist-my-own-bot/398299
Cloudflare Community
How to whitelist my own bot
Our website is getting a massive amount of bot traffic which is overwhelming the backend server. So I recently enabled “Super Bot Fight Mode”, to issue a managed challenge to “Definitely Automated” traffic. But I now want to run my own bots against the site, to do things like latency measurement, and detecting and removing cached wordpress adm...