MobileMoxie Glossary
Crawling
Crawling is the process that Google and other search engines use to find information to include in their index. When search engines crawl a site, they are using links to find pages, and then they act as a ‘headless browser’, basically a browser without a screen, to capture the most important information on a page, that would be displayed for anyone with a screen. Google’s shift to Mobile-First Indexing might actually be more accurately described as ‘Mobile-First Crawling’ if the only thing that changed was the crawler, which previously was simulating a desktop computer, but with this change, they began crawling with a bot that simulated a mobile browser instead. Crawlers follow links within a site, but also links from one site to another, and links included in XML sitemaps. Crawlers also seem to follow rel=alternate style links which can be used for bi-directional annotation, HREF Lang, and canonicals.
Related Terms: