robots.txt is not valid
Can anyone help me how to fix this issue?
3 Replies
robots.txt isn't supposed to be a HTML file
Google for Developers
Robots.txt Introduction and Guide | Google Search Central | Docum...
Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.
Actually, the wikipedia article is probably better as an intro https://en.wikipedia.org/wiki/Robots.txt
Robots.txt
robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
The standard, developed in 1994, relies on voluntary compliance. Malicious bots can use the file as a directory of which pag...