Google introduced a brand new robots.txt report inside Google Search Console and on the similar time mentioned it would sundown the outdated robots.txt tester software. The brand new software exhibits which robots.txt recordsdata Google discovered for the highest 20 hosts in your web site, the final time they have been crawled, and any warnings or errors encountered.
Google additionally mentioned on X, “We’re additionally making related information out there from the Web page indexing report. As a part of this replace, we’re sunsetting the robots.txt tester.”
To entry this, go to the settings hyperlink to your Search Console profile or click on right here. That is what it appears to be like like:
Here’s what the report appears to be like like for this web site, sure, I’ve a vital error. Listed below are these particulars and observe, you’ll be able to request recrawl of your robots.txt file:
This assist doc goes by way of much more about this report.
Some are usually not pleased that the robots.txt tester software is gone however Bing nonetheless has one:
Good options to Google’s depreciated software to validate your robots.txt:
• Merkle: https://t.co/33yMW9IlPB
• Ryte: https://t.co/jYAGuHN2q5
• Screaming Frog (in-crawl): https://t.co/zsr2KMiNmC— Cyrus website positioning (@CyrusShepard) November 15, 2023
I really like the brand new robots.txt reporting (it is in settings btw), however eradicating the robots.txt Tester? I really like that software and it may be super-helpful. There are different instruments that do comparable issues, together with one from Bing, however I might like to see that keep in GSC. 🙂 https://t.co/hMIJQjfpQV
— Glenn Gabe (@glenngabe) November 15, 2023
Discussion board dialogue at X.
