How to create and send a Robots.txt file to Google for your site

If you use the hosting service for your site, for example, Wix or Blogger, you may not have to edit the Robots.txt file directly. Instead, your provider can provide a search settings page or other mechanisms for indicating search engines, which pages of the site can be indexed and which ones are not.

If you want to hide or open one of the pages from search engines, read the instructions for changing the page visibility at your hosting, for example, with a search query “How to hide a page from search engines in WIX”. This will help you control which files can be available for bypassing search robots using the Robots.txt file.

Fundamentals of creating and using the Robots.txt file for your Google site

The Robots.txt file is located in the root of your site. For example, for www.example.com, the Robots.txt file will be located at www.example.com/robots.txt. This is a regular text file that follows the Robots Exclusion Standard Exception standard.

The Robots.txt file consists of one or more directives, each of which blocks or allows access for a particular robot to the specified path on the site. By default, all files on the site are allowed for bypass, unless otherwise specified.

An example of the Robots.txt base file for your Google site

Here is an example of a simple Robots.txt file with two rules:

User-agent: Googlebot
Disallow: /nogooglebot/
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

What does this file mean:

The Googlebot robot cannot bypass the URL starting with https://example.com/nogooglebot/.
All other robots can bypass the entire site. This rule could be omitted, since by default all robots can bypass the site.
The site uses a site card that is located at https://www.example.com/sitemap.xml.

How to create and test a Robots.txt file for your Google website

Creating a Robots.txt file and its testing consists of several stages:

Create a file named Robots.txt.
Add the relevant search rules to the file.
Download the Robots.txt file to the root of your site.
Check the availability of the file for search engines and its correctness.

How to write rules for rabots.txt on your Google website

The rules in the Robots.txt file are designed to specify robots which parts of your site can be bypassed. Here are a few recommendations for writing the rules:

Each rule begins with a user-agent indication, which determines for which search robot the rules are applied.
Each rule may also contain Disallow (for a ban on bypass) and Allow (to resolve bypass). For example:

User-agent: Googlebot
Disallow: /private/
User-agent: *
Allow: /

Robots.txt testing for your Google site

After downloading the Robots.txt file to the site, be sure to test that it is available for search robots. To do this, open the incognito window in your browser and go to your Robots.txt, for example, https://example.com/robots.txt. If you see the contents of your file, it means that it is available for processing.

To test and eliminate problems with markings, you can use:

Robots.txt report on Google Search Console (if the file is available on your site).
Using the Google library for testing local files Robots.txt.

How to send a Robots.txt file to Google

After the Robots.txt file is uploaded and tested, Google robots will automatically find it and start using it. There is no need to manually send the file. However, if you have updated the file and want Google to quickly upgrade it in the cache, you can use the Robots.txt update function in the Google Search Console.

Useful Robots.txt rules for your Google site

Here are some useful examples of the Robots.txt file:

To prohibit the bypass of the entire site:

 User-agent: *
 Disallow: /

To prohibit a detour of a particular directory:

 User-agent: *
 Disallow: /private/

To resolve bypass only to a particular robot:

 User-agent: Googlebot
 Allow: /
 User-agent: *
 Disallow: /

Contacts for clarifications on Robots.txt and SEO for your site in Google

If you have questions about setting up the Robots.txt file or other SEO issues, you can contact our SEO companion info@seo.computer or through WhatsApp by number +79202044461.

ID: 4