A crawler, also known as a spider or a bot, is the software Google uses to process and index the content of webpages. The AdSense crawler visits your site to determine its content in order to provide relevant ads.
Here are some important facts to know about the AdSense crawler:
The crawler report is updated weekly
The crawl is performed automatically and Google is not able to accommodate requests for more frequent crawling.
The AdSense crawler is different from the Google crawler
The two crawlers are separate, but they do share a cache. Google does this to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Search Console crawler is separate.
Resolving AdSense crawl issues will not resolve issues with the Google crawl
Resolving the issues listed on your Crawler access page will have no impact on your placement within Google search results.
The crawler indexes by URL
Google crawler will access site.com and www.site.com separately. However, Google crawler will not count site.com and site.com/#anchor separately.
The crawler won’t access pages or directories prohibited by a robots.txt file
Both the Google and AdSense Mediapartners crawlers honor your robots.txt file. If your robot.txt file prohibits access to certain pages or directories, then they will not be crawled.
Note that if you’re serving ads on pages that are being roboted out with the line User-agent: *, then the AdSense crawler will still crawl these pages. To prevent the AdSense crawler from accessing your pages, you need to specify User-agent: Mediapartners-Google in your robots.txt file.
The crawler will attempt to access URLs only where Google ad tags are implemented
Only pages displaying Google ads should be sending requests to its systems and being crawled.
The crawler will attempt to access pages that redirect
When you have “original pages” that redirect to other pages, Google crawler must access the original pages to determine that a redirect is in place. Therefore, Google crawler’s visit to the original pages will appear in your access logs.
At this time, Google is unable to control how often its crawlers index the content on your site. Crawling is done automatically by Google bots. If you make changes to a page, it may take up to 1 or 2 weeks before the changes are reflected in Google index.