Introduction to Robots.txt for your site in Google

The Robots.txt file tells search engines which URLs on your site can be available for their indexation. This file is used primarily in order to avoid overloading your site with requests; It is not a means to exclude a web page from the results of the Google search. If you want to exclude a page from the search results in Google, use the Noindex methag or protect the page with a password.

If you use CMS, such as Wix or Blogger, you may not need (or impossible) to edit the Robots.txt file directly. Instead, your CMS can provide a search settings page or another mechanism to specify search engines, whether your page should be scanned.

If you want to hide or open one of your pages for the search engines, look for instructions on changing the page visibility in search engines in your CMS (for example, look for "WIX Hide a page from search engines").

What is a Robots.txt file and how is it used for your Google site?

The Robots.txt file is primarily used to control the traffic of search engines to your site, and usually in order to exclude the file from the Google indexation, depending on the type of file:

How Robots.txt affects different types of files for your Google site

Web pages

You can use the Robots.txt file for web pages (, PDF or other formats that Google can read) to control the indexation traffic if you think your server will be overloaded with requests from the Google search robot, or to avoid the pre-long or duplicate pages on your site.

Warning: Do not use the Robots.txt file as a way to hide a web page (including PDF and other text formats supported by Google) from the results of the Google search.

If other pages refer to your page with descriptive text, Google can index the URL without visiting the page. If you want to completely exclude the page from the search results, use other methods, such as password protection or Noindex metatheg.

If your web page is blocked in the Robots.txt file, its URL can still appear in the search results, but the description will not be displayed in the results. Media files, videos, PDF and other files built into the blocked page will also not be scanned if they are not mentioned on other pages that are allowed for indexation. If you see this search result for your page and want to fix it, delete Robots.txt. If you want to hide the page completely from the search in Google, use another method.

Media files

You can use the Robots.txt file to control the traffic indexation and to prevent the appearance of images, video and audio files in the results of the Google search. This will not prevent links to your media files from other pages or from users.

Learn more about how to prevent the appearance of images in Google.
Learn more about how to delete or limit video files from the appearance in Google.

Resource files

You can use the Robots.txt file to block resource files, such as insignificant images, scripts or styles, if you think that pages without these resources will not be significantly changed. However, if the absence of these resources complicates the indexation of the Google robot, do not block them, otherwise Google will not be able to correctly analyze the pages that depend on these resources.

Understanding the restrictions on the Robots.txt file for your Google site

Before creating or editing the Robots.txt file, you should understand the restrictions on this URL locking method. Depending on your goals and the situation, you may need to consider other mechanisms to make sure that your URLs will not be found on the Internet.

Robots.txt file rules may not be supported by all search engines. Instructions in the Robots.txt file cannot forcibly affect the behavior of robots; It depends on whether the robot should follow these instructions. Although Googlebot and other respected web crauls follow the Robots.txt file rules, other kralers may not do this. Therefore, if you want to save security information from web crauls, it is better to use other locking methods, for example, protect the files with a password.
Different crawles can interpret the syntax in different ways. Despite the fact that dear web crauls follow the rules of the Robots.txt file, each kraler can interpret these rules differently. You should know the correct syntax for referring to different web crawers, as some of them may not understand certain instructions.
The page prohibited in the Robots.txt file can still be indexed if there are links from other sites to it. Although Google will not scan and index the content blocked in Robots.txt, he can still find and index the forbidden URL if there are links from other pages on the Internet. As a result, the URL address and, possibly, other publicly available data, such as the text of links to the page, can still appear in the results of the Google search. To completely prevent the appearance of your URL in the results of the Google search, use password protection, Noindex methag or delete the page at all.

Attention: Combining several rules for kraling and indexation can cause a conflict between the rules. Find out how to combine kraling rules with indexing and display.

Creation or update of the Robots.txt file for your Google site

If you decide that you need a Robots.txt file, find out how to create it. If the file already exists, find out how to update it.

Recommended information for your site in Google

Creation and Sending the Robots.txt file to Google

The Robots.txt file should be in the root of your site. Find out how to create a Robots.txt file, see examples and study the Robots.txt file rules.

Sending the updated Robots.txt file to Google

Using the Robots.txt report, you can easily check whether Google can process your Robots.txt files. Follow these steps to send updated Robots.txt files to Google.

How Google interprets the specification of the Robots.txt file

Explore the details of the various rules for the Robots.txt file and how Google interprets the specification of the Robots.txt file.

If you have questions about the Robots.txt file for your Google site or other aspects of SEO, you can contact the SEO company "Seo.computer" by email: info@seo.computer or through WhatsApp: +79202044461.

ID 25