Problems with crawlbots
Hi Guys,
We have a problem with a website of ours. The website is super big 25,000 pages approximately but because of this we get super many crawl requests. Is there any way to mitigate this via cloudflare. We don't want to turn them off completely.
1 Reply
How do you mean?