Join three of Reddit’s top executives in this exclusive AMA (Ask Me Anything) to discover how you can tap into Reddit’s unique platform to drive brand growth.
Maximize your SEO efforts in 2024 with insights on Google’s SGE, algorithm updates, and expert tips to keep your site ahead.
Download this guide and learn how to optimize and manage Google Performance Max campaigns, with expert insights and actionable strategies to ensure your campaigns are effective.
Join us in analyzing 3 case studies that show the importance of driving brand search behavior and engagement, and how to do it in months, instead of years.
Join three of Reddit’s top executives in this exclusive AMA (Ask Me Anything) to discover how you can tap into Reddit’s unique platform to drive brand growth.
Join us as we dive into exclusive survey data from industry-leading SEOs, digital marketers, content marketers, and more to highlight the top priorities and challenges that will shape the future of search in 2025.
Google’s Gary Illyes reveals Googlebot collects links instead of “following” them, contradicting official documentation.
In a recent episode of Google’s Search Off The Record podcast, Analyst Gary Illyes clarified how Googlebot interacts with links during the crawling process.
His insight contradicts the widely held belief that Googlebot navigates websites by following links in real-time.
According to Illyes, the Googlebot gathers links for later processing rather than following them in a linear fashion.
This misunderstanding stems from Google’s own documentation.
“It’s my pet peeve,” Illyes stated during the podcast, referring to Google’s support pages.
He continues:
“On our site, we keep saying Googlebot is following links, but no, it’s not following links. It’s collecting links, and then it goes back to those links.”
Google’s official documentation on crawlers states (emphasis mine):
“Crawler (sometimes also called a ‘robot’ or ‘spider’) is a generic term for any program that is used to automatically discover and scan websites by following links from one web page to another.”
The document implies that Googlebot navigates the web by actively following links in real-time.
This highlights a discrepancy between Google’s public messaging and the actual functionality of their crawler, raising questions about other potential misunderstandings in the SEO community.
This revelation has several potential implications for how we understand Google’s crawling process:
Many SEO strategies are built on the assumption that Googlebot traverses websites by following internal links like a visitor on a page.
If Illyes’ description is accurate, it suggests that Googlebot’s behavior is more complex than previously understood.
While this revelation doesn’t invalidate current SEO best practices, it does highlight the need for SEO professionals to stay informed about the nuances of how Google operates.
Hear Illyes’ statement in the podcast below:
Featured Image: Ribkhan/Shutterstock
Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s degree in communications, …
Conquer your day with daily search marketing news.
Join Our Newsletter.
Get your daily dose of search know-how.
In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses — and careers.
Copyright © 2024 Search Engine Journal. All rights reserved. Published by Alpha Brand Media.