How to confirm that Googlebot and other Google search robots actually go to your site through Google

You can check whether the web robot that turns to your server is a real Googlebot. This is useful if you are worried that spammers or other attackers can access your site, pretending to be Googlebot.

Google robots are divided into three categories:

Google robots, check on your Google website

Description of Google robots that can visit your site, as well as their corresponding IP bands:

Ordinary Google robots on your Google website

These are robots used for Google products, such as Googlebot. These robots always follow Robots.txt rules for automatic bypasses.

Examples of domains: CRAWL-***-***-***-***. Googlebot.com or geo-crawl-***-***-***-***. GEO.googlebot.com

Example Json File: Googlebot.json

Special Google robots on your Google website

These robots perform specific functions for Google products (for example, Adsbot) when there is an agreement between the website and the product on the process of bypass. These robots can or cannot follow Robots.txt rules.

Example: Rate-Limited-Proxy-***-***-***-***. Google.com, Special-crawlers.json

Robots activated by the user on your site on Google

These tools and products of products where the end user activates the data request. For example, Google Site Verifier, which is launched at the user's request. These robots ignore Robots.txt rules.

Example: ***-***-***-***. Gae.googleusercontent.com or google-proxy-***-***-***-***. Google.com.

Googlebot testing methods on your Google website

There are two ways to check whether the Googlebot web robot is:

Manual method: For single checks, use command tools. This method is suitable for most cases.
Automatic way: For mass inspections, use an automatic solution to compare the IP address of the robot with published IP-bands Googlebot.

Using command tools for checking Googlebot on your Google website

Run the reverse DNS request to the IP address addressing your server using the HOST command.
Make sure that the domain name corresponds to one of the following: Googlebot.com, Google.com or Googleusercontent.com.
Run the straight DNS-request to the domain name obtained in the previous step using the Host command.
Make sure that the domain name corresponds to the original IP address, which was recorded in your logs.

Example 1:

host 66.249.66.11
66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Example 2:

host 35.247.243.240
240.243.247.35.in-addr.arpa domain name pointer geo-crawl-35-247-243-240.geo.googlebot.com.
host geo-crawl-35-247-243-240.geo.googlebot.com
geo-crawl-35-247-243-240.geo.googlebot.com has address 35.247.243.240

Example 3:

host 66.249.90.77
77.90.249.66.in-addr.arpa domain name pointer rate-limited-proxy-66-249-90-77.google.com.
host rate-limited-proxy-66-249-90-77.google.com
rate-limited-proxy-66-249-90-77.google.com has address 66.249.90.77

Using automatic solutions to check Googlebot on your Google website

You can also identify Googlebot by IP address by comparing it with published IP addresses for Googlebot and other Google robots:

Ordinary robots such as Googlebot
Special robots such as Adsbot
Requests activated by the user (users)
Requests activated by the user (Google)

For other Google IP addresses, from which access to your site (for example, for applications) can be made, compare the IP address with the general list of Google IP addresses. Please note that the IP address in JSON files are presented in CIDR format.

If you have questions or help, you can contact the SEO company "SEO.COMPUTER" by email info@seo.computer Or whatsApp +79202044461.

ID 41