Crawlers returning 404, but pages are working
Hi, we're getting a spike of 404 not found errors in Google Search Console but when I manually open the urls they work and if I manually request indexing for any of those it will index without any issues.
How can I identify where crawlers return 404? Since we're using Cloudflare between our hosting provider so I'm hinting that could be the lead.
5 Replies
Not found (404)
You check within the last 24 hours who has been blocked.
It's possible you have a rule blocking the ASN or the IPs. You can obtain Google's crawler IPs/ASN/known headers to search through the blocked and determine what rule got them.
Cloudflare doesn't throw 404's though your site would.
So maybe something to do with the server and how it's configured.
I figured that a lot of the pages were actually indexed tonight so I surely wasn't blocked and still showing on google. Looking further those are actually pages that don't exist.
I had some errors with robots.txt before and wrong sitemap link within (including www after https) and I believe it took some time before that was recrawled and used.
Yeah bad pages would do it