Join this leadership discussion for proven techniques to build long-term relationships and keep your clients coming back.
Discover the latest trends, tips, and strategies in SEO and PPC marketing. Our curated articles offer in-depth analysis, practical advice, and actionable insights to elevate your digital marketing efforts.
In this guide, industry experts explore the challenges and opportunities presented by the onslaught of AI tools, and provide actionable tips for thriving amid these shifts.
Discover the latest trends, tips, and strategies in SEO and PPC marketing. Our curated articles offer in-depth analysis, practical advice, and actionable insights to elevate your digital marketing efforts.
Join three of Reddit’s top executives in this exclusive AMA (Ask Me Anything) to discover how you can tap into Reddit’s unique platform to drive brand growth.
Join this leadership discussion for proven techniques to build long-term relationships and keep your clients coming back.
Google explains how its search engine crawls the web in latest “How Search Works” video.
Google has published a fresh installment of its educational video series “How Search Works,” explaining how its search engine discovers and accesses web pages through crawling.
In the seven-minute episode hosted by Google Analyst Gary Illyes, the company provides an in-depth look at the technical aspects of how Googlebot—the software Google uses to crawl the web—functions.
Illyes outlines the steps Googlebot takes to find new and updated content across the internet’s trillions of webpages and make them searchable on Google.
Illyes explains:
“Most new URLs Google discovers are from other known pages that Google previously crawled.
You can think about a news site with different category pages that then link out to individual news articles.
Google can discover most published articles by revisiting the Category page every now and then and extracting the URLs that lead to the articles.”
Googlebot starts by following links from known webpages to uncover new URLs, a process called URL discovery.
It avoids overloading sites by crawling each one at a unique, customized speed based on server response times and content quality.
Googlebot renders pages using a current version of the Chrome browser to execute any JavaScript and correctly display dynamic content loaded by scripts. It also only crawls publicly available pages, not those behind logins.
Related: Google Answers A Crawl Budget Issue Question
Illyes highlighted the usefulness of sitemaps—XML files that list a site’s URLs—to help Google find and crawl new content.
He advised developers to have their content management systems automatically generate sitemaps.
Optimizing technical SEO factors like site architecture, speed, and crawl directives can also improve crawlability.
Here are some additional tactics for making your site more crawlable:
The latest video comes after Google launched the educational “How Search Works” series last week to shed light on the search and indexing processes.
The newly released episode on crawling provides insight into one of the search engine’s most fundamental operations.
In the coming months, Google will produce additional episodes exploring topics like indexing, quality evaluation, and search refinements.
The series is available on the Google Search Central YouTube channel.
Google’s crawling process, as outlined in their recent “How Search Works” series episode, involves the following key steps:
Marketers can adopt the following strategies to enhance their content’s discoverability and crawlability for Googlebot:
Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s degree in communications, …
Conquer your day with daily search marketing news.
Join Our Newsletter.
Get your daily dose of search know-how.
In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses — and careers.
Copyright © 2024 Search Engine Journal. All rights reserved. Published by Alpha Brand Media.