John Mueller from Google answered a question about crawling errors displayed by Google Search Console (GSC) on well-loaded pages in the browser.
John Mueller explained that the problem is usually on the server side, not Googlebot.
The person who asked the question tried to verify the page in Google Search Console (GSC), but the page was stubbornly not verified.
It’s as if something is blocking Googlebot and displaying an error message.
John Mueller claims that if GSC displays server errors when Googlebot crawls a webpage, then they do exist and are not Google’s errors.
Also read: SEO tools for beginners
Temporary Crawl Errors
Mueller then pointed out that some of these problems are temporary.
Sometimes these errors are actually temporary; for example, the server may be down for maintenance, there may be a DNS issue that caused part of the Internet to shut down, or the server may be overloaded and prevent tracking.
Problems encountered by the person who asked the question If the page loads normally but is always verified with GSC (or try to use one of Google’s tools for testing, such as a rich search results tester or a mobile-friendly tester) and fails the verification, then It is a problem with the server.
Diagnosing Googlebot Crawl Errors
If your IP is shared by other sites on the server, you can use diagnostic techniques to determine whether it is a server-wide configuration problem.
What can be done is to identify the IP address where the site is located, and then run the ISP through a reverse IP checker, which can display other sites hosted on that IP address.
It then takes the list of domains and runs it through one of Google’s tools, such as the AMP checker or Google’s rich results checker tool.
Report error responses for one or more domains, so this may indicate a server-wide error.
Each server has a server log, which is a good place to start diagnosing the cause of the problem.
These server logs will show the exact date and time when the error occurred, as well as the IP address of the visitor who caused the error.
A typical problem might be a misconfigured firewall, which may be too strict and prevent Google.
If you don’t have access to server logs, you should call the web hosting customer service as recommended by John Mueller.