Leave a request
This article will be useful for those whose work includes scanning projects, as well as setting parameters that are suitable for both medium-sized sites with up to 500 thousand URLs, and for more compact web resources.
In our company, as in any large digital agency, an important aspect is the optimization and automation of routine processes. This frees up time for more strategic tasks.
After I had been at the agency for two months, I was assigned to scan a few projects every month. I'm not new to SEO, but my main specialization is account management and marketing.
From time to time, as an account manager, I analyzed the visibility of client sites in search results, and also did a detailed scan of web resources. To do this, I used the Screaming Frog SEO Spider program. Having installed the software, I received a license from the management and did not touch the program settings, since everything worked well, and as you know, you should not change something that is not broken.
However, everything changed when I was assigned to lead a larger project with 430 thousand URLs. Previously, I worked on projects with a maximum of 1700 URLs. The problem was not that scanning the project took a day (which, as it turned out, is the norm), but that when starting the scan, the program often froze at 88-94% of the process.
Although the instructions for Screaming Frog are quite detailed, they do not provide specific recommendations for setting up the program for large projects. Then I started looking for information on how to properly use the SEO Spider Tool. After re-reading several articles, I realized that they repeat the same thing as in the instructions, only more briefly.
As a result, I decided to turn to experienced colleagues for help, who, for which I thank them very much, explained to me how the program works and gave me several useful recommendations for setting it up.
After three days of experimentation, trial and error, and consultation with colleagues, I came to the following conclusions:
Device characteristics:
As you can see, the characteristics of the device are average and quite suitable for performing my tasks. I didn’t need powerful hardware, since this is quite enough for everyday tasks.
Screaming Frog SEO Spider program settings (version 14.1):
Important! In the program settings, be sure to specify that project information is written to an SSD disk; this significantly speeds up the scanning process.
With such device characteristics and the specified settings, scanning a project with 420-450 thousand URLs takes about 30 hours, if at least 12 hours of scanning are carried out in parallel with other tasks.
The decision to write and publish this article came after unsuccessful attempts to find recommendations for setting up the Screaming Frog program in open sources. Most of the articles only explained what and how the program works, but did not give practical advice on setting it up for large projects. I hope this information is useful. Read, use, share with colleagues, and may the Force be with you!
If you have any questions, you can contact the SEO studio "SEO COMPUTER" for any question by email info@seo.computer.
ID 1744
