{"id":638586,"date":"2024-09-16T14:47:54","date_gmt":"2024-09-16T11:47:54","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/google-updates-crawlers-and-user-triggered-fetchers-documentation\/"},"modified":"2024-09-16T14:47:54","modified_gmt":"2024-09-16T11:47:54","slug":"google-updates-crawlers-and-user-triggered-fetchers-documentation","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/google-updates-crawlers-and-user-triggered-fetchers-documentation\/","title":{"rendered":"#Google updates crawlers and user-triggered fetchers documentation"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a26e15f8dde5\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a26e15f8dde5\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/google-updates-crawlers-and-user-triggered-fetchers-documentation\/#With_that_Google_expanded_how_Google_products_are_affected_by_each_crawler_and_gave_robotstxt_examples_for_each_crawler\" >With that, Google expanded how Google products are affected by each crawler and gave robots.txt examples for each crawler.<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"subhead\" itemprop=\"alternativeHeadline\"><span class=\"ez-toc-section\" id=\"With_that_Google_expanded_how_Google_products_are_affected_by_each_crawler_and_gave_robotstxt_examples_for_each_crawler\"><\/span>With that, Google expanded how Google products are affected by each crawler and gave robots.txt examples for each crawler.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><\/p>\n<div class=\"bialty-container\">\nGoogle made a <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/watch-movies-tv-seriess\/\" data-internallinksmanager029f6b8e52c=\"8\" title=\"Watch Movies &amp; TV Series\" target=\"_blank\" rel=\"noopener\">series<\/a> of updates to its <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/overview-google-crawlers\">crawlers and user-triggered fetchers<\/a> documentation, mostly breaking out the single-page document into multiple pages and documents. But Google also expanded what product each crawler affects with a new section next to each crawler and added a robots.txt snippet for each crawler to demonstrate how to use the user agent tokens by each crawler.<\/p>\n<p><strong>What Google said. <\/strong>Google <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/developers.google.com\/search\/updates#reorganizing-the-crawler-documentation\">posted<\/a> about these changes saying:<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>\n\u201cReorganized the documentation for Google\u2019s crawlers and user-triggered fetchers. We also added explicit notes about what product each crawler affects, and added a robots.txt snippet for each crawler to demonstrate how to use the user agent tokens. There were no meaningful changes to the content otherwise.\u201d<\/p>\n<p>\u201cThe documentation grew very long which limited our ability to extend the content about our crawlers and user-triggered fetchers.\u201d\n<\/p><\/blockquote>\n<p><strong>What\u2019s new. <\/strong>Besides for just moving a lot of content around, to organize the content. Google also added the \u201caffected products\u201d sections and also the \u201cExample robots.txt group\u201d sections. Here is a screenshot of this <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>lied to the Googlebot crawler, but it was added to each individual crawler:<\/p>\n<figure class=\"wp-block-image size-full\"><img fetchpriority=\"high\" decoding=\"async\" width=\"2046\" height=\"1520\" alt=\"Google Crawler Affected Products Example Robots Group 1726479983\" class=\"wp-image-446596\" srcset=\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983.png 2046w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-455x338.png.webp 455w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-800x594.png.webp 800w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-152x113.png.webp 152w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-768x571.png.webp 768w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-1536x1141.png 1536w\" data-lazy-sizes=\"(max-width: 2046px) 100vw, 2046px\" src=\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983.png\"><img fetchpriority=\"high\" decoding=\"async\" width=\"2046\" height=\"1520\" src=\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983.png\" alt=\"Google Crawler Affected Products Example Robots Group 1726479983\" class=\"wp-image-446596\" srcset=\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983.png 2046w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-455x338.png.webp 455w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-800x594.png.webp 800w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-152x113.png.webp 152w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-768x571.png.webp 768w,https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-crawler-affected-products-example-robots-group-1726479983-1536x1141.png 1536w\" sizes=\"(max-width: 2046px) 100vw, 2046px\"><\/figure>\n<p>I <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.seroundtable.com\/google-updates-its-google-crawlers-and-fetchers-documentation-38073.html\">pulled out<\/a> each one for you:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Googlebot:<\/strong> Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/news\/\" data-internallinksmanager029f6b8e52c=\"2\" title=\"News\" target=\"_blank\" rel=\"noopener\">News<\/a>, and Discover.<\/li>\n<li><strong>Googlebot Image:<\/strong> Crawling preferences addressed to the Googlebot-Image user agent affect Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented.<\/li>\n<li><strong>Googlebot Video:<\/strong> Crawling preferences addressed to the Googlebot-Video user agent affect video-related Google Search features and other products dependent on videos.<\/li>\n<li><strong>Googlebot News:<\/strong> Crawling preferences addressed to the Googlebot-News user agent affect all surfaces of Google News (for example, the News tab in Google Search and the Google News app).<\/li>\n<li><strong>Google StoreBot:<\/strong> Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping).<\/li>\n<li><strong>Google-InspectionTool:<\/strong> Crawling preferences addressed to the Storebot-Google user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products.<\/li>\n<li><strong>GoogleOther:<\/strong> Crawling preferences addressed to the GoogleOther user agent don\u2019t affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. It has no effect on Google Search or other products.<\/li>\n<li><strong>GoogleOther-Image: <\/strong>Crawling preferences addressed to the GoogleOther-Image user agent don\u2019t affect any specific product, similar to GoogleOther. GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs.<\/li>\n<li><strong>GoogleOther-Video:<\/strong> Crawling preferences addressed to the GoogleOther-Video user agent don\u2019t affect any specific product, similar to GoogleOther. GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs.<\/li>\n<li><strong>Google-CloudVertexBot: <\/strong>Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners\u2019 for building Vertex AI Agents. It has no effect on Google Search or other products.<\/li>\n<li><strong>Google-Extended:<\/strong> Google-Extended is a standalone product token that web publishers can use to manage whether their sites help improve Gemini Apps and Vertex AI generative APIs, including future generations of models that power those products. Google-Extended does not impact a site\u2019s inclusion or ranking in Google Search.<\/li>\n<li><strong>APIs-Google: <\/strong>Crawling preferences addressed to the APIs-Google user agent affect the delivery of push notification messages by Google APIs.<\/li>\n<li><strong>AdsBot Mobile Web:<\/strong> Crawling preferences addressed to the AdsBot-Google-Mobile user agent affect Google Ads\u2019 ability to check web page ad quality.<\/li>\n<li><strong>AdsBot:<\/strong> Crawling preferences addressed to the AdsBot-Google user agent affect Google Ads\u2019 ability to check web page ad quality.<\/li>\n<li><strong>AdSense:<\/strong> Crawling preferences addressed to the <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">Media<\/a>partners-Google user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads.<\/li>\n<li><strong>Google-Safety: <\/strong>The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it\u2019s unaffected by crawling preferences.<\/li>\n<li><strong>Feedfetcher: <\/strong>Feedfetcher is used for crawling RSS or Atom feeds for Google News and PubSubHubbub.<\/li>\n<li><strong>Google Publisher Center:<\/strong> Google Publisher Center fetches and processes feeds that publishers explicitly supplied for use in Google News landing pages.<\/li>\n<li><strong>Google Read Aloud: <\/strong>Upon user request, Google Read Aloud fetches and reads out web pages using text-to-speech (TTS).<\/li>\n<li><strong>Google Site Verifier:<\/strong> Google Site Verifier fetches Search Console verification tokens.<\/li>\n<\/ul>\n<p><strong>Why we care. <\/strong>Reading through these affected product sections may help you better understand how each crawler affects various aspects of Google. Some don\u2019t impact Google Search at all, while others are fundamental to how Google Search works.<\/p>\n<p>Also the new robots.txt examples may be very useful to you and your development teams.\n<\/p><\/div>\n<p><\/p>\n<div class=\"about-author\">\n    About the author<\/p>\n<div class=\"information\">\n<div class=\"author-module\">\n<div class=\"row\">\n<div class=\"col-12 col-lg-3 text-center\">\n<div class=\"avatar\">\n                        <img loading=\"lazy\" decoding=\"async\" class=\"img-fluid rounded-circle avatar-border\" alt=\"Barry Schwartz\" width=\"140\" height=\"140\" src=\"https:\/\/searchengineland.com\/images\/authors\/BarrySchwartz-lg.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"img-fluid rounded-circle avatar-border\" src=\"https:\/\/searchengineland.com\/images\/authors\/BarrySchwartz-lg.jpg\" alt=\"Barry Schwartz\" width=\"140\" height=\"140\">\n                                            <\/div>\n<\/p><\/div>\n<div class=\"col-12 col-lg-9\">\n<div class=\"about\">\n<div class=\"name\">\n                            <strong>Barry Schwartz<\/strong>\n                        <\/div>\n<div class=\"row g-2 pt-2\">\n<div class=\"col-auto twitter\">\n                                    <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/twitter.com\/intent\/follow?original_referer=https%3A%2F%2Fsearchengineland.com%2F&amp;region=follow_link&amp;screen_name=rustybrick&amp;tw_p=followbutton&amp;variant=2.0\" aria-label=\"opens in a new tab\"><i class=\"fab fa-x-twitter\"><\/i><\/a>\n                            <\/div>\n<div class=\"col-auto\">\n                                    <a rel=\"nofollow noopener\" target=\"_blank\" href=\"http:\/\/www.linkedin.com\/in\/rustybrick\" aria-label=\"opens in a new tab\"><i class=\"fab fa-linkedin\"><\/i><\/a>\n                                <\/div>\n<\/p><\/div>\n<p>                        Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.rustybrick.com\/\">RustyBrick<\/a>, a NY based web consulting firm. He also runs <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.seroundtable.com\/\">Search Engine Roundtable<\/a>, a popular search blog on very advanced SEM topics.<\/p>\n<p>In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the &#8220;US Search Personality Of The Year,&#8221; you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O&#8217;Clock.<\/p>\n<p>Barry can be followed\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/twitter.com\/rustybrick\/\">on X here<\/a>\u00a0and you can learn more about\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.rustybrick.com\/barry\">Barry Schwartz over here<\/a> or on his <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.barryschwartz.org\/\">personal site<\/a>.                 <\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/searchengineland.com\/google-updates-crawlers-and-user-triggered-fetchers-documentation-446594\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>With that, Google expanded how Google products are affected by each crawler and gave robots.txt examples for each crawler. Google made a series of updates to its crawlers and user-triggered fetchers documentation, mostly breaking out the single-page document into multiple pages and documents. But Google also expanded what product each crawler affects with a new&#8230;<\/p>\n","protected":false},"author":1,"featured_media":638587,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/09\/google-spider-web-1920.jpg","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[147868,78070],"class_list":["post-638586","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-google-seo","tag-seo"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/638586","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=638586"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/638586\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/638587"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=638586"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=638586"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=638586"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}