.Google's John Mueller responded to an inquiry regarding why Google.com indexes pages that are actually prohibited from crawling through robots.txt as well as why the it is actually safe to dismiss the relevant Browse Console records regarding those creeps.Crawler Traffic To Question Criterion URLs.The person inquiring the concern recorded that robots were actually developing hyperlinks to non-existent inquiry criterion URLs (? q= xyz) to web pages with noindex meta tags that are actually additionally shut out in robots.txt. What caused the inquiry is actually that Google is creeping the hyperlinks to those webpages, receiving shut out by robots.txt (without watching a noindex robots meta tag) after that receiving reported in Google.com Look Console as "Indexed, though blocked by robots.txt.".The person inquired the adhering to question:." But right here is actually the big inquiry: why will Google index web pages when they can not also observe the information? What is actually the benefit in that?".Google's John Mueller verified that if they can't creep the web page they can not view the noindex meta tag. He likewise creates an intriguing reference of the website: hunt driver, advising to dismiss the results due to the fact that the "normal" individuals will not view those end results.He created:." Yes, you're proper: if our experts can't crawl the webpage, we can not see the noindex. That claimed, if our experts can not crawl the web pages, after that there's certainly not a whole lot for our team to index. So while you may see some of those pages along with a targeted website:- concern, the common customer won't find them, so I wouldn't fuss over it. Noindex is additionally alright (without robots.txt disallow), it only means the URLs will certainly end up being actually crept (and wind up in the Search Console document for crawled/not indexed-- neither of these conditions trigger problems to the rest of the site). The fundamental part is that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the constraints in using the Web site: hunt evolved search operator for analysis reasons. Among those causes is considering that it is actually not connected to the frequent hunt mark, it is actually a distinct point completely.Google.com's John Mueller talked about the web site hunt driver in 2021:." The brief response is that a web site: question is actually not indicated to become complete, nor made use of for diagnostics objectives.A web site concern is actually a specific kind of search that limits the end results to a certain website. It is actually primarily just words web site, a colon, and afterwards the web site's domain name.This inquiry restricts the results to a particular website. It is actually not indicated to become an extensive compilation of all the web pages from that site.".2. Noindex tag without making use of a robots.txt is actually great for these sort of circumstances where a robot is linking to non-existent web pages that are getting discovered through Googlebot.3. URLs along with the noindex tag will certainly generate a "crawled/not catalogued" entry in Browse Console and that those won't have a negative effect on the rest of the internet site.Check out the inquiry and respond to on LinkedIn:.Why would Google.com index webpages when they can not even find the material?Included Image through Shutterstock/Krakenimages. com.