Leave a request
General chorelers of the site are used to search for information and create Google search engines. They are also used for specific tasks of products and analysts. These crawlers always follow Robots.txt rules with automatic bypass. Technical characteristics of the main kralers of the site apply to general kraler.
As a rule, the general chulets of the site work with the IP addresses listed in the special googlebot.json facility, and the reverse DNS checks to their hostemams correspond to CRAWL-***-***-***-***. GoogleBot.com or ***-***-***. GEO-CRAWL-***-***-***-***. GEO.googlebot.com.
The list shows the general huts of the site, their user-agent lines in HTTP checks, corresponding to user-agent tokens for Robots.txt directives, as well as products that are influenced by Crutting settings for each kraler. Some kralers have several user-agent tokens-it is enough to compare one of them to apply the rules. The list is not exhaustive, it includes only the most common requests and those on which questions came.
Attention: the user-agent line in the HTTP request can be faked. It is recommended to check whether the visitor is really a chuler of the Google search engine site.
Lines of User-Agent in HTTP checks for Googlebot have two main types-for smartphones and for desktop devices. Example for smartphones: Mozilla/5.0 (Linux; Android) Applewebkit/... Chrome/... Mobile Safari/... (Compatible; Googlebot/2.1; ...). For desktop devices - Mozilla/5.0 Applewebkit/... (Compatible; Googlebot/2.1; ...).
Less commonly there are options such as Mozilla/5.0 (Compatible; Googlebot/2.1; ...) or just Googlebot/2.1 (...).
Robots.txt uses Googlebot token to manage this crawler of the site.
GoogleBot settings affect Google search products, including search, Discover, image and news search, video and news.
Line User-Agent: Googlebot-Image/1.0.
Token in Robots.txt: Googlebot-Image.
Crauling Management by this kraler of the site is reflected in the search for images, Discover, video content and display of logos and phavicons in the results of the Google search.
The string user-agent: Googlebot-Video/1.0.
Token in Robots.txt: Googlebot-video.
This crauls of the site affects the functions of searching for video and products related to video content.
This crawler of the site does not use a separate HTTP User-Agent. Crauling of news content is performed using different Googlebot user-agent lines.
Token in Robots.txt: Googlebot-news.
Crowling settings affect Google news services, including news and mobile applications.
Lines of User-Agent are for desktop and mobile devices indicating Storebot-Google.
Token in Robots.txt: Storebot- Google.
This site is used to collect data for trading products, such as the purchases section in searching for Google.
The USR-Agent lines for desktop and mobile devices contain Google-infectionTool.
Token at Robots.txt: Google-inspectionTool.
This site is used to test search results and does not affect the general results of Google search.
Lines User-Agent: Mozilla/... (Compatible; Googleometer) for mobile and desktop devices.
Token at Robots.txt: Googleother.
This site is used for various single or internal tasks, without affecting the results of Google search.
USER-Agent line: Googleometer-Image/1.0.
Token in Robots.txt: Googleometer-Image.
Crowler of the site is optimized for collecting images without affecting specific Google products.
USER-Agent line: Googleometer-Video/1.0.
Token in Robots.txt: Googleometer-Video.
Used to collect video files without affecting the search results.
The user-agent line contains Google-Cloudvertexbot.
Token in Robots.txt: Google- Cloudvertexbot.
It is used for kraling related to the construction of AI-agents and does not affect the search results of Google.
It does not have a separate HTTP string user-agent. Token in Robots.txt: Google- Extended.
Allows the owners of the site to control the use of content for training AI models without affecting the ranking in the search for Google.
The designation Chrome/W.x.y.z in the user-agent lines is a template indicating the version of the Chrome browser used by the crauls of the site. The version number is updated over time.
When searching or filtering by user-agent in logs, it is recommended to use substitution signs for the version instead of an accurate number.
For any questions, you can contact SEO SEO.computer by email info@seo.computer Or through WhatsApp +79202044461.
ID 141
