birkelund
CDCloudflare Developers
•Created by birkelund on 9/12/2024 in #general-help
CNAME blocked in cloudflare
5 replies
CDCloudflare Developers
•Created by birkelund on 8/6/2024 in #general-help
Image transformation not working
2 replies
CDCloudflare Developers
•Created by birkelund on 7/25/2024 in #general-help
Cloudflare blocks Google from reading robots.txt
I have a strange situation.
We see across many of our sites on cloudflare, that Google is no longer able og reading the robots.txt file. This leads googles crawler to start crawling alot of urls which it shouldnt.
When we open Google Search Console for any domain, we see that Google sees the robots.txt file as being empty. There is a manual functionality for triggering google to read the robots.txt again. When we do this we get "An error has occurred. Please try again later.". Then we can see that google shows that it consideres the robots.txt to be empty.
When we disable proxy in cloudflare and tries, then it works fine, and Google can read the robots.txt file.
The strange thing is that it seems like this started happening suddenly about 2 or 3 weeks ago on many of our sites.
I have checked the following settings
- No custom WAF rules
- No useragent blocking created
- Bot fight mode is disabled
- Block AI scrapers and Crawlers is disabled
- Security Level is Medium
- Browser Integrity check is disabled
When this happens, no events shows up in the Security>>Events.
Anybody who have experienced anything similar or have any idea why Cloudflare would cause the failure at Google?
3 replies
CDCloudflare Developers
•Created by birkelund on 10/23/2023 in #general-help
Extra ACME TXT records
Last year, we had multiple domains that had extra TXT records which preventet the generation of a letsencrypt certificate on the host server.
https://community.cloudflare.com/t/extra-acme-txt-records-preventing-renewal/412449
After much back and forth, cloudflare deleted the records on all the domains we report.
Now I notice that there is again 2 of our domains that have these phantom txt records, however this time when i send a mail to support@cloudflare.com, it get a reply back saying that tickets can no longer be generated through the mail, and cloudflare only allow tickets for billing, account, and registrar issues.
This is an issues in cloudflares end, that only they can solve.
Anybody who can guide me. How am I to go about getting in touch with cloudflare?
12 replies