A detailed guide on how Google search works for your site

Google Search is a fully automated search engine that uses software known as web crauls that regularly explore the Internet to find pages that can be added to the index. In fact, most of the pages that are displayed in the search results were not manually sent for indexation, but were found and added automatically when studying the Internet with web crauls. This document explains in detail how the search in Google works in the context of your site. Knowing this process will help you correct scan errors, get pages in the index and optimize the site display in searching for Google.

Looking for something less technical? Check out the site How Search Workswho explains how the search works from the point of view of the user.

A few comments before starting work with Google for your site

Before plunging into the details of the search engine, it is important to note that Google does not accept payment for a more frequent scan of the site or for higher positions in the search results. If someone claims the opposite, know that this is not true.

Google does not guarantee that the site will be scanned, indexed or displayed in search results, even if its pages correspond to Google Search Essentials.

Three stages of Google search work for your site

Search in Google takes place in three stages, and not all pages go through each of them:

• Kraling: Google uploads text, images and videos from the found pages on the Internet using automatic programs called kraler.
• Indexing: Google analyzes the text, images and video materials on the page, and then saves this information in the Google index - a huge database.
• Search results: When the user searches for Google, the system returns information that, according to Google, is the most relevant to user's request.

Crauling in Google: How your site can be found for searching in Google

The first stage is the search for pages on the Internet. Since there is no central registry of all web pages, Google is constantly looking for new and updated pages to add them to its list of famous pages. This process is called the "search for the URL." Some pages are already known because Google has already visited them. Other pages are found when Google extracts a link from a well -known page to a new one. For example, if the site category refers to a new article on the blog. You can also send a list of pages (site map) so that Google can index them.

As soon as Google finds the URL page, he can visit this page to understand what it is contained. For this, a large number of computers are used that scan billions of pages on the Internet. The program that performs this task is called Googlebot (also known as Crauler, Robot, Bot or Spider). Googlebot uses the algorithm to determine which pages to scan, how often and how many pages you need to download from each site. Googlebot is also tuned in such a way as not to overload the servers of sites, scanning them too often. This mechanism depends on the response of servers (for example, HTTP 500 errors indicate that you need to slow down the process).

However, Googlebot does not always scan all the pages that it finds. Some pages may not be available to Kraler because of the site settings, for example, due to a ban on scanning through the Robots.txt file or due to the need to authorize for access.

During Crailingin, Google also renders the page and performs JavaScript, using the current version of the Chrome browser, which helps to understand which content is displayed on the page. This is important, since many sites depend on JavaScript to display content, and without rendering Google may not see important data.

Indexing: How Google analyzes the content of your site

After the page was scanned, Google is trying to understand what this page is talking about. This stage is called indexing, and it includes the processing and analysis of text content, as well as metathegs, such as tags