Understanding Google's Web Crawling Strategy: Key Learnings from Analyst Gary Illyes

Google Analyst Gary Illyes recently announced on LinkedIn that his goal for the year is to reduce the frequency of web crawling. This announcement follows a Reddit discussion where users observed that Google's crawling activity has decreased compared to previous years.

Illyes mentioned that Google is still crawling a similar amount. However, he highlighted the importance of improving scheduling and prioritizing URLs that are deemed more worthy of crawling.

Illyes’ statement supports the current conversation in the SEO community regarding the idea of a “crawl budget.” This concept suggests that websites need to adhere to a specific number of pages that search engines can crawl each day in order to index their pages.

Recently, Google’s Search Relations team dispelled this misunderstanding in a podcast, clarifying how Google determines crawling priorities based on different factors.

Crawling Prioritization & Search Demand

In a podcast published two weeks ago, Illyes explained how Google decides how much to crawl:

If search demand decreases, the crawl limit will also decrease.

The term "search demand" likely refers to the number of searches for a specific topic from Google's point of view. In simpler terms, if people are searching less for a certain topic, Google may not crawl websites related to that topic as often.

Illyes highlighted the significance of demonstrating to search engines that a website's content is valuable. He mentioned that in order to improve crawling frequency, it is essential to persuade search engines that the content is worth retrieving. This is essentially what the scheduler pays attention to.

Although Illyes didn’t elaborate on how to achieve this, one interpretation could be to ensure that content remains relevant to user trends and stays up to date.

Focus On Quality

Google previously clarified that a fixed “crawl budget” is largely a myth.

Instead, the search engine’s crawling decisions are dynamic and driven by content quality.

Illyes explained that scheduling is constantly changing. Once they receive signals from search indexing indicating that the content quality has improved on a certain number of URLs, they will increase the demand.

The Way Forward

Illyes’ mission to improve crawling efficiency by reducing the amount of crawling and bytes on wire is a step towards a more sustainable and practical web.

Illyes is asking for suggestions from the community on any fascinating internet drafts or standards from IETF or other standards bodies that could help with this project.

He believes that reducing crawling without compromising crawl-quality would be advantageous for everyone.

Why SEJ Cares

When Illyes talks about reducing crawling, it reminds us of the importance of quality and relevance in SEO. It's not just about technical stuff, but also about making content that is valuable and meets what users are searching for.

By understanding the dynamic nature of Google’s crawling decisions, we can all make more informed choices when optimizing our websites and allocating resources.

How This Can Help You

With the insights provided by Illyes, here are some practical steps you can follow:

First and foremost, make quality your top priority. Concentrate on producing top-notch, useful, and captivating content that meets the needs of users and matches the current search trends.

Regularly updating and refreshing your content is important to keep it valuable to your target audience. It shows that you are staying current and relevant.

It is also crucial to monitor search demand trends and adapt your content strategy accordingly. This will help address emerging topics and trends, ensuring that your website remains relevant and worthy of being crawled by search engines.

Make sure your website is following technical best practices by having a clean, well-structured architecture and a strong internal linking strategy. This will help search engines crawl and index your site more efficiently.

Don't forget to consider the important points shared by Illyes and the insights provided by Google's Search Relations team as you continue to improve your SEO strategies.

With these insights, you’ll be equipped to succeed if and when Google reduces crawling frequency.

Featured Image: Skorzewiak/Shutterstock

Editor's P/S:

Google's announcement to reduce web crawling frequency has sparked discussions within the SEO community. The concept of a "crawl budget" has been dispelled, emphasizing that Google's crawling decisions are dynamic and prioritize high-quality content. By focusing on creating valuable, relevant content that meets user search demands, website owners can improve their chances of being crawled by search engines.

Illyes' initiative to improve crawling efficiency highlights the importance of sustainability in web crawling. Reducing the frequency and data consumption will lead to a more efficient web, benefiting both search engines and website owners. By understanding Google's crawling prioritization and search demand factors, SEOs can adapt their strategies to ensure their websites remain visible and valuable in the search landscape.