.Google's John Mueller answered a concern regarding why Google.com marks pages that are refused from crawling through robots.txt and why the it is actually secure to disregard the similar Look Console files about those crawls.Crawler Visitor Traffic To Query Specification URLs.The individual asking the concern chronicled that crawlers were creating web links to non-existent query guideline Links (? q= xyz) to webpages with noindex meta tags that are actually additionally blocked out in robots.txt. What prompted the inquiry is that Google.com is actually creeping the web links to those pages, getting blocked out by robots.txt (without envisioning a noindex robotics meta tag) after that getting shown up in Google.com Look Console as "Indexed, though shut out by robots.txt.".The person asked the complying with inquiry:." But listed below is actually the large concern: why would Google mark web pages when they can't also observe the web content? What is actually the advantage during that?".Google's John Mueller confirmed that if they can not creep the page they can't view the noindex meta tag. He also creates an exciting acknowledgment of the site: hunt operator, advising to overlook the results because the "ordinary" customers will not observe those results.He created:." Yes, you're correct: if our company can not crawl the page, we can not view the noindex. That claimed, if our experts can't crawl the web pages, after that there is actually not a lot for our company to index. So while you might view several of those webpages with a targeted site:- inquiry, the average user will not observe all of them, so I would not fuss over it. Noindex is actually also fine (without robots.txt disallow), it merely suggests the Links are going to wind up being crept (as well as end up in the Browse Console record for crawled/not catalogued-- neither of these statuses trigger problems to the remainder of the website). The essential part is that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the restrictions in operation the Website: hunt progressed search operator for analysis factors. Some of those reasons is actually considering that it is actually not hooked up to the frequent hunt mark, it is actually a distinct thing completely.Google's John Mueller talked about the internet site search driver in 2021:." The quick solution is that a website: concern is actually not indicated to become full, nor utilized for diagnostics purposes.A website query is actually a details type of hunt that limits the outcomes to a certain web site. It is actually basically only the word web site, a bowel, and after that the web site's domain.This query confines the outcomes to a specific website. It's certainly not indicated to be an extensive collection of all the pages from that internet site.".2. Noindex tag without using a robots.txt is alright for these sort of situations where a bot is actually connecting to non-existent pages that are receiving uncovered by Googlebot.3. Links with the noindex tag will produce a "crawled/not listed" item in Browse Console and that those won't possess a damaging impact on the rest of the site.Go through the concern and also address on LinkedIn:.Why would Google.com mark web pages when they can't also observe the content?Featured Picture by Shutterstock/Krakenimages. com.