Home SEO Google’s Robots FAQs Has Been Eliminated

Google’s Robots FAQs Has Been Eliminated

Google’s Robots FAQs Has Been Eliminated


Google Robot Asking Question

Earlier this week, Google eliminated its Robots.txt FAQ assist doc from its search developer documentation. When requested, John Mueller from Google replied to Alexis Rylko saying, “We replace the documentation infrequently. Be at liberty to submit suggestions in the event you really feel one thing’s lacking. Robots.txt is unquestionably nonetheless a factor.”

The Robots FAQ doc lived over right here: builders.google.com/search/docs/crawling-indexing/robots/robots-faq

That now redirects to the principle Google robots.txt assist web page.

What did the Robots FAQ web page say, effectively the Wayback Machine has a replica, so I’ll archive it right here:

(Q) Does my web site want a robots.txt file?

(A) No. When Googlebot visits an internet site, we first ask for permission to crawl by making an attempt to retrieve the robots.txt file. An internet site with out a robots.txt file, robots meta tag, or X-Robots-Tag HTTP headers will usually be crawled and listed usually.

(Q) Which methodology ought to I take advantage of to dam crawlers?

(A) It relies upon. Briefly, there are good causes to make use of every of those strategies:

  • robots.txt: Use it if crawling of your content material is inflicting points in your server. For instance, you might need to disallow crawling of infinite calendar scripts. Do not use the robots.txt to dam personal content material (use server-side authentication as a substitute), or deal with canonicalization. To ensure that a URL shouldn’t be listed, use the robots meta tag or X-Robots-Tag HTTP header as a substitute.
  • robots meta tag: Use it if it’s worthwhile to management how a person HTML web page is proven in search outcomes or to ensure that it is not proven.
  • X-Robots-Tag HTTP header: Use it if it’s worthwhile to management how content material is proven in search outcomes or to ensure that it is not proven.

(Q) Can I take advantage of robots.txt, robots meta tag, or the X-Robots-Tag HTTP header to take away another person’s web site from search outcomes?

(A) No. These strategies are solely relevant to websites the place you possibly can modify the code or add information. Study extra about easy methods to take away info from Google.

(Q) How can I decelerate Google’s crawling of my web site?

(A) You’ll be able to usually modify the crawl charge setting in your Google Search Console account.

(Q) I take advantage of the identical robots.txt for a number of web sites. Can I take advantage of a full URL as a substitute of a relative path?

(A) No. The principles within the robots.txt file (with exception of sitemap:) are solely legitimate for relative paths.

(Q) Can I place the robots.txt file in a subdirectory?

(A) No. The file have to be positioned within the topmost listing of the web site.

(Q) I need to block a non-public folder. Can I forestall different individuals from studying my robots.txt file?

(A) No. The robots.txt file could also be learn by varied customers. If folders or filenames of content material aren’t meant for the general public, do not checklist them within the robots.txt file. It isn’t advisable to serve completely different robots.txt information primarily based on the consumer agent or different attributes.

(Q) Do I’ve to incorporate an enable rule to permit crawling?

(A) No, you do not want to incorporate an enable rule. All URLs are implicitly allowed and the enable rule is used to override disallow guidelines in the identical robots.txt file.

(Q) What occurs if I’ve a mistake in my robots.txt file or use an unsupported rule?

(A) Internet crawlers are usually very versatile and sometimes won’t be swayed by minor errors within the robots.txt file. On the whole, the worst that may occur is that incorrect or unsupported guidelines shall be ignored. Keep in mind although that Google cannot learn minds when deciphering a robots.txt file; now we have to interpret the robots.txt file we fetched. That stated, if you’re conscious of issues in your robots.txt file, they’re often simple to repair.

(Q) What program ought to I take advantage of to create a robots.txt file?

(A) You need to use something that creates a legitimate textual content file. Widespread applications used to create robots.txt information are Notepad, TextEdit, vi, or emacs. Learn extra about creating robots.txt information. After creating your file, validate it utilizing the robots.txt Tester.

(Q) If I block Google from crawling a web page utilizing a robots.txt disallow rule, will it disappear from search outcomes?

(A) Blocking Google from crawling a web page is more likely to take away the web page from Google’s index.

Nevertheless, robots.txt disallow doesn’t assure {that a} web page won’t seem in outcomes: Google should still determine, primarily based on exterior info akin to incoming hyperlinks, that it’s related and present the URL within the outcomes. When you want to explicitly block a web page from being listed, use the noindex robots meta tag or X-Robots-Tag HTTP header. On this case, do not disallow the web page in robots.txt, as a result of the web page have to be crawled to ensure that the tag to be seen and obeyed. Learn to management what you share with Google

(Q) How lengthy will it take for modifications in my robots.txt file to have an effect on my search outcomes?

(A) First, the cache of the robots.txt file have to be refreshed (we usually cache the contents for as much as at some point). You’ll be able to velocity up this course of by submitting your up to date robots.txt to Google. Even after discovering the change, crawling and indexing is an advanced course of that may typically take fairly a while for particular person URLs, so it is not possible to provide an actual timeline. Additionally, needless to say even when your robots.txt file is disallowing entry to a URL, that URL might stay seen in search outcomes regardless of that proven fact that we will not crawl it. When you want to expedite elimination of the pages you’ve got blocked from Google, submit a elimination request.

(Q) How can I quickly droop all crawling of my web site?

(A) You’ll be able to quickly droop all crawling by returning a 503 (service unavailable) HTTP standing code for all URLs, together with the robots.txt file. The robots.txt file shall be retried periodically till it may be accessed once more. We don’t advocate altering your robots.txt file to disallow crawling.

(Q) My server shouldn’t be case-sensitive. How can I disallow crawling of some folders utterly?

(A) Guidelines within the robots.txt file are case-sensitive. On this case, it is strongly recommended to ensure that just one model of the URL is listed utilizing canonicalization strategies. Doing this lets you have fewer traces in your robots.txt file, so it is simpler so that you can handle it. If this is not potential, we advisable that you simply checklist the frequent combos of the folder title, or to shorten it as a lot as potential, utilizing solely the primary few characters as a substitute of the total title. For example, as a substitute of itemizing all higher and lower-case permutations of /MyPrivateFolder, you may checklist the permutations of /MyP (if you’re sure that no different, crawlable URLs exist with these first characters). Alternately, it could make sense to make use of a robots meta tag or X-Robots-Tag HTTP header as a substitute, if crawling shouldn’t be a difficulty.

(Q) I return 403 Forbidden for all URLs, together with the robots.txt file. Why is the location nonetheless being crawled?

(A) The 403 Forbidden HTTP standing code, in addition to different 4xx HTTP standing codes, is interpreted because the robots.txt file does not exist. Which means that crawlers will usually assume that they will crawl all URLs of the web site. In an effort to block crawling of the web site, the robots.txt have to be returned with a 200 OK HTTP standing code, and should comprise an acceptable disallow rule.

(Q) Is the robots meta tag a alternative for the robots.txt file?

(A) No. The robots.txt file controls which pages are accessed. The robots meta tag controls whether or not a web page is listed, however to see this tag the web page must be crawled. If crawling a web page is problematic (for instance, if the web page causes a excessive load on the server), use the robots.txt file. If it’s only a matter of whether or not or not a web page is proven in search outcomes, you should utilize the robots meta tag.

(Q) Can the robots meta tag be used to dam part of a web page from being listed?

(A) No, the robots meta tag is a page-level setting.

(Q) Can I take advantage of the robots meta tag exterior of a


(A) No, the robots meta tag must be within the

part of a web page.

(Q) Does the robots meta tag disallow crawling?

(A) No. Even when the robots meta tag at present says noindex, we’ll have to recrawl that URL sometimes to verify if the meta tag has modified.

(Q) How does the nofollow robots meta tag evaluate to the rel=”nofollow” hyperlink attribute?

(A) The nofollow robots meta tag applies to all hyperlinks on a web page. The rel=”nofollow” hyperlink attribute solely applies to particular hyperlinks on a web page. For extra info on the rel=”nofollow” hyperlink attribute, see our documentation on user-generated spam and the rel=”nofollow”.

(Q) How can I verify the X-Robots-Tag for a URL?

(A) A easy method to view the server headers is to make use of the URL Inspection Device function in Google Search Console. To verify the response headers of any URL, attempt looking for “server header checker”.

I assume possibly Google thinks it’s redundant from what’s already printed on the opposite pages?

Discussion board dialogue at X.


Supply hyperlink


Please enter your comment!
Please enter your name here