Leave a request
Screaming Frog SEO Spider is a powerful website analysis tool that helps solve many SEO problems. However, sometimes problems may arise when trying to collect data from some web resources, for example, due to blocking by User Agent. In such cases, you can try changing the User Agent in the program itself to bypass such restrictions.
To change the User Agent, open the Screaming Frog program and follow a few simple steps.
1. Open Screaming Frog, go to the menu and click on the button Upload at the top of the interface. Then select the option Enter Manuals.
2. In the window that appears, enter the URL of the site you want to analyze.
3. If you see zeros or a scan error in the scan results (for example, error code 609), this may mean that the site is blocking requests from the standard User Agent. In this case, you need to change it.
4. To do this, go to the menu Configuration and select User Agent from the drop-down list.
5. After this, a window with available User Agents will open. Select one of the suggested options, for example, Google or Yandex bot.
6. After selecting the desired User Agent, the HTTP request data will be updated automatically. Confirm your selection to save your changes.
7. Now try loading the site again for analysis using the new User Agent via Upload And Enter Manuals.
As a rule, after changing the User Agent, the program will successfully collect the data available for the selected bot. This may include meta tags and content that is hidden from regular site visitors.
If the data is still not displayed, you can try choosing another bot from the suggested list for more efficient parsing.
If you have any questions about using SEO tools or data parsing, you can contact the SEO studio "SEO COMPUTER" with any questions by email info@seo.computer.
ID 6304
