Basic setup of an SEO parser: step-by-step instructions

Parsing websites can be time-consuming, especially if you use generic tools. But many of them can be customized for specific tasks to make the process more efficient. Let's look at how to do this using the example of a popular SEO parser.

Data storage location;User Agent;Parsing speed;Parsing sections;What to parse;Account access;robots.txt;Saving settings.

Setting up data storage location

The first step in setting up a parser is choosing a data storage location. To do this, go to the "File" menu and select "Settings". Select "Data storage type" from the drop-down list. There are two options available in this section:

In RAM. This is suitable for quick downloads if you have enough RAM but limited hard drive space. However, this may not be enough for large sites.

On your hard drive. In this case, you choose a path to store the data. The process will be slower, but the files will take up less RAM.

Then go back to Settings and select "Memory Allocation" to specify the amount of RAM the system can use. This is useful if you have other tasks running on your computer in parallel.

Setting up User Agent

Next you need to configure the User Agent. Go to the "Configuration" menu and select "User-Agent". Here you can configure the user agent that will be used when parsing sites.

You can choose standard agents, for example, for mobile devices, so that the parser disguises itself as a bot and receives complete information without being blocked by the site.

Setting parsing speed

Now we set up the parsing speed. Go to the "Configuration" menu and select "Speed". Here you can specify the number of threads that will be used when downloading data. For weak computers, it is optimal to choose from 3 to 5 threads.

Setting up sections for parsing

If you need to audit only certain sections of the site, configure them in the "Configurations" section through the "Include" item. Here you can enter the path to the desired partition to check whether it will be included in the parsing process.

If a section is not included in the selection, a notification about this will appear. To exclude sections from parsing, use the "Exclude" item.

Setting up Spider Crawl

Go to "Configuration", select "Spider" and configure the data scanning type. Here you can choose exactly what data will be scanned, disabling unnecessary types of information.

If the site is not too large, you can leave the default settings and crawl all available data.

Connecting a Google account

To search for problematic pages, connect your Google account. Go to "Configuration", then "API Access" and select Google Search Console. This will allow you to quickly find pages that are not linking to and fix the problem.

Setting up work with robots.txt

Next, choose how the parser should work with the robots.txt file. In "Configuration" select one of the following options:

Ignore – if you need to analyze the entire site;

Respect – if the parser must comply with the rules of robots.txt;

Ignore robots.txt but report status – if you need to receive data about the status of pages.

Saving settings

If you use the same settings frequently, it is recommended to save them as default. To do this, in the "Configuration" menu, select "Save Current Configuration as Default". You can also create multiple profiles for different tasks and quickly switch between them.

If you have any questions or need help setting up SEO tools, write to the SEO studio "SEO COMPUTER" by email info@seo.computer.

ID 4473