Google employee Gary Ilsh confirmed on Twitter that the company is working to improve robots.txt checking. This tool is currently only available in the older version of Search Console.
This information appeared in response to a complaint about a possible bug in the work of robots.txt or incorrect reading of this file by the service.
Gary Ilsh tweeted, “We re in the process of integrating the tester with the production parser,†adding that he thinks the problem lies with the tool itself.
January 6, 2021
In early 2019, Google announced that it did not plan to disable its robots.txt checker. The company promised to eventually migrate the tool to the new version of Search Console, however, judging by Ilsh s answer, this is not an urgent task for Google.
Source: Search Engine Roundtable
Article rating