Leave a request
Googlebot is a common name for two types of web robots used in Google Search:
You can determine the Googlebot subtype by looking at the USR-Agent's HTTP heading. However, both types of robots are subordinate to the same product token (user-agent token) in Robots.txt, so you can not choose one of them (Googlebot Smartphone or Googlebot Desktop) using Robots.txt.
For most sites, Google Search first of all indexes the mobile version of content. Therefore, most requests from Googlebot will be made using a mobile robot, and the minority - using a desktop robot.
For most Googlebot sites, your site should not visit your site more than once every few seconds on average. However, due to delays, it can be expected that this indicator will be slightly higher in short time intervals. If your site does not have time to cope with Googlebot requests, you can reduce the speed of it bypass.
Googlebot can scan the first 15 MB -files or supported text file. Each resource that refers to, such as CSS or JavaScript, is loaded separately, and each request is limited by the same limit in size of the file. After the first 15 MB, the file stops scanning, and only these 15 MB of the file are transmitted for indexing. The limit in size is applied to incompressed data. Other Google robots, such as Googlebot Video and Googlebot Image, may have other restrictions.
When traveling from the IP addresses in the USA, the Googlebot time zone is a Pacific time.
Other technical characteristics of Googlebot are described in the Google robot review.
Googlebot finds new URLs for a detour primarily through links built into the already visited pages. It is almost impossible to hide the site without publishing links to it. For example, as soon as someone crosses the link from your “secret” site to another site, the URL of your “secret” site may appear in Referrer tag and will be preserved and published by another site in its logs.
If you want to prevent Googlebot bypass on your site, you have several options. Remember that there is a difference between the bypass and indexation: Googlebot blocking from going around the page will not prevent the URL of this page to appear in search results:
Googlebot blocking affects Google Search (including Discover and all Google Search functions), as well as other products such as Google Images, Google Video and Google News.
Before you decide to block Googlebot, it is important to understand that the USR-Agent HTTP-head used by Googlebot is often faked by other robots. Therefore, it is important to check that the request really comes from Google. The best way to confirm that the request comes from Googlebot is to execute a reverse DNS post on the initial IP address of the request or to monitor the IP address with the GoogleBot IP addresses.
If you have questions about SEO or you need a consultation, you can contact our SEO companion through email info@seo.computer or WhatsApp: +79202044461.
ID 59
