mapple
CDCloudflare Developers
•Created by mapple on 5/10/2023 in #pages-help
robots.txt
resolved - thanks very much for your help - for anyone in the future reading this double check response with curl https://www.yoursite.com/robots.txt
10 replies
CDCloudflare Developers
•Created by mapple on 5/10/2023 in #pages-help
robots.txt
I've carried on digging around - am I going to need something like proxy flare to serve the robots.txt? https://developers.cloudflare.com/pages/platform/functions/plugins/community-plugins/
10 replies
CDCloudflare Developers
•Created by mapple on 5/10/2023 in #pages-help
robots.txt
thanks that's really useful - I've added a 404 and it's all behaving better - I'm still getting the index page served on the custom domain site.com when I try and view site.com/robots.txt .... (I have added robots.txt to the root folder I upload to deploy and can see it there in the uploaded files) how can I get pages to not deisplay the index page if googlebot comes looking for robots.txt?
10 replies