Google's Latest Update: Enhancements to Google-Extended Crawler Documentation
Discover the recent updates to Google's Google-Extended web crawler documentation, shedding light on changes in product naming and the implications for search. Dive into the details of controlling content access for AI model training.
Introduction to Google-Extended User Agent
In a recent development, Google rolled out updates to its Google-Extended web crawler user agent documentation, unveiling significant changes in product nomenclature and providing insights into the effects on search results. This update has sparked interest among website owners and digital marketers as it addresses concerns related to blocking the Google-Extended crawler.
The revamped documentation now offers more comprehensive guidance on managing content access, especially for leveraging AI model training. This move signifies Google's commitment to enhancing user experience and ensuring transparency in its web crawling processes.
Exploring Google-Extended User Agent Features
Introduced on a significant date, September 28, 2023, Google-Extended presents web publishers with a unique user agent tool for regulating the crawling behavior on their websites. Through the implementation of the Robots Exclusion Protocol, publishers gain the ability to permit or restrict the Google-Extended user agent, empowering them to control the inclusion of their content in AI training datasets.
Google characterizes Google-Extended as a 'standalone product token,' a departure from conventional user agent terminology, which might require a deeper understanding from website administrators and content creators. The initial announcement highlighted the utility of the new user agent in enhancing AI models, particularly those powering Bard and Vertex AI generative APIs.
Insights from Google Changelog and Updates
Google maintains a detailed changelog to document essential modifications in guidance and communication channels with web publishers and the broader search marketing community. The most recent update in the Google-Extended documentation aligns with the rebranding of Bard to Gemini Apps, elucidating the connection between Google-Extended indexing and Gemini Apps and Vertex AI generative APIs.
The clarification provided in the updated documentation aims to allay concerns regarding the impact of opting out of Google-Extended AI data collection on search outcomes. Notably, the revised guidance emphasizes that Google-Extended crawling is specific to Gemini Apps and does not influence Google Search results, ensuring transparency and addressing potential misconceptions.
