{"id":646178,"date":"2024-12-07T07:00:38","date_gmt":"2024-12-07T04:00:38","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/crawl-budget-what-you-need-to-know-in-2025\/"},"modified":"2024-12-07T07:00:38","modified_gmt":"2024-12-07T04:00:38","slug":"crawl-budget-what-you-need-to-know-in-2025","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/","title":{"rendered":"#Crawl budget: What you need to know in 2025"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a23d4eee33ec\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a23d4eee33ec\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Confused_about_crawl_budget_This_comprehensive_guide_explains_it_all_%E2%80%93_from_server_capacity_to_fixing_crawling_issues_for_better_indexing\" >Confused about crawl budget? This comprehensive guide explains it all \u2013 from server capacity to fixing crawling issues for better indexing.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Why_would_search_bots_limit_crawling\" >Why would search bots limit crawling?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#The_cost_of_crawling\" >The cost of crawling<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#What_is_crawl_budget\" >What is crawl budget?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#What_causes_issues_with_crawl_budget\" >What causes issues with crawl budget<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Why_crawl_budget_is_important\" >Why crawl budget is important<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#How_crawl_budget_issues_happen\" >How crawl budget issues happen<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Quality\" >Quality<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Volume\" >Volume<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Accessibility\" >Accessibility<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#How_to_identify_crawl_budget_problems\" >How to identify crawl budget problems<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#See_what_the_search_engines_are_reporting\" >See what the search engines are reporting<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Log_files\" >Log files<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#How_to_fix_crawl_budget_problems\" >How to fix crawl budget problems<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Another_word_of_warning\" >Another word of warning<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Fixing_crawl_budget_issues_through_the_robotstxt\" >Fixing crawl budget issues through the robots.txt<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Improving_the_quality_and_load_speed_of_pages\" >Improving the quality and load speed of pages<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Control_crawling_through_robotstxt\" >Control crawling through robots.txt<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Consider_nofollow_links_on_internal_links\" >Consider nofollow links on internal links<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/buradabiliyorum.com\/en\/crawl-budget-what-you-need-to-know-in-2025\/#Navigating_crawl_budget_for_SEO_success_in_2025\" >Navigating crawl budget for SEO success in 2025<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"subhead\" itemprop=\"alternativeHeadline\"><span class=\"ez-toc-section\" id=\"Confused_about_crawl_budget_This_comprehensive_guide_explains_it_all_%E2%80%93_from_server_capacity_to_fixing_crawling_issues_for_better_indexing\"><\/span>Confused about crawl budget? This comprehensive guide explains it all \u2013 from server capacity to fixing crawling issues for better indexing.<br \/>\n<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><\/p>\n<div class=\"bialty-container\">\n<p>Crawl budget is a common source of concern and confusion in SEO.\u00a0<\/p>\n<p>This guide will explain everything you need to know about crawl budget and how it may impact your technical SEO efforts in 2025.<\/p>\n<h2 class=\"wp-block-heading\" id=\"h-why-would-search-bots-limit-crawling\"><span class=\"ez-toc-section\" id=\"Why_would_search_bots_limit_crawling\"><\/span>Why would search bots limit crawling?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Google\u2019s Gary Illyes <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/developers.google.com\/search\/blog\/2017\/01\/what-crawl-budget-means-for-googlebot\" target=\"_blank\" rel=\"noopener\">provided an excellent explanation<\/a> about crawl budget, describing how Googlebot strives to be a \u201c<a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/developers.google.com\/search\/blog\/2017\/01\/what-crawl-budget-means-for-googlebot#crawl-rate-limit\" target=\"_blank\" rel=\"noopener\">good citizen of the web<\/a>.\u201d This principle is key to understanding the concept and why it exists.<\/p>\n<p>Think of when you last saw tickets to your favorite band go on sale.\u00a0<\/p>\n<p>Too many users flood the website, overwhelming the server and causing it not to respond as intended. This is frustrating and often prevents users from buying tickets.<\/p>\n<p>This can also h<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>en with bots. Remember when you forgot to adjust the crawling speed or number of simultaneous connections allowed on your favorite site crawler and brought down the website you were crawling on?\u00a0<\/p>\n<p>Googlebot could also do this. It could hit a website too frequently or through too many \u201cparallel connections\u201d and cause the same effect, essentially overwhelming the server.\u00a0<\/p>\n<p>As a \u201cgood citizen,\u201d it is designed to avoid that happening.<\/p>\n<p>Google sets its \u201ccrawl capacity limit\u201d for a site based on what the site can handle.\u00a0<\/p>\n<p>If the site responds well to the crawl, it will continue at that pace and increase the volume of connections.\u00a0<\/p>\n<p>If it responds poorly, then the speed of fetching and connections used will be lowered.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-the-cost-of-crawling\"><span class=\"ez-toc-section\" id=\"The_cost_of_crawling\"><\/span>The cost of crawling<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Crawling, parsing and rendering use up resources, and there are financial considerations involved in the process.<\/p>\n<p>Yes, that\u2019s one reason Google and other search engines may adjust how they crawl a site to benefit it.\u00a0<\/p>\n<p>However, I imagine some financial cost calculation goes into determining how frequently a URL should be crawled.<\/p>\n<h2 class=\"wp-block-heading\" id=\"h-what-is-crawl-budget\"><span class=\"ez-toc-section\" id=\"What_is_crawl_budget\"><\/span>What is crawl budget?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Crawl budget refers to the amount of time and resources Googlebot allocates to crawling a website. It is determined by two key factors: the crawl capacity limit and crawl demand.\u00a0<\/p>\n<ul class=\"wp-block-list\">\n<li>The crawl capacity limit reflects how much crawling a site can handle without performance issues.<\/li>\n<li>Crawl demand is based on Googlebot\u2019s assessment of the website\u2019s content, including individual URLs, and the need to update its understanding of those pages.<\/li>\n<\/ul>\n<p>More popular pages are crawled more frequently to ensure the index remains up-to-date. <\/p>\n<p>Google <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget#in-sum\" target=\"_blank\" rel=\"noopener\">calculates this budget<\/a> to balance the resources it can afford to spend on crawling with the need to protect both the website and its own infrastructure.<\/p>\n<h2 class=\"wp-block-heading\" id=\"h-what-causes-issues-with-crawl-budget\"><span class=\"ez-toc-section\" id=\"What_causes_issues_with_crawl_budget\"><\/span>What causes issues with crawl budget<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Not all sites will ever notice any impact of having a crawl budget.\u00a0<\/p>\n<p>Google clearly says only<a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget#who-this-guide-is-for\" target=\"_blank\" rel=\"noopener\"> three types of websites<\/a> need to manage their crawl budget actively. These are:<\/p>\n<ul class=\"wp-block-list\">\n<li>Large sites (i.e., those with over 1 million unique pages).<\/li>\n<li>Medium or large sites with frequently updating content.<\/li>\n<li>Sites with a high volume of \u201cDiscovered \u2013 currently not indexed\u201d as detailed in Google Search Console\u2019s Page Indexing report. \u00a0 \u00a0 \u00a0 \u00a0 \u00a0<\/li>\n<\/ul>\n<p>Now, I would advise caution before dismissing your website as none of the above: crawl your site.<\/p>\n<p>You may feel that your small ecommerce store only has a couple of thousand SKUs and a handful of informational pages.\u00a0<\/p>\n<p>In reality, though, with faceted navigation and pagination, you may have ten times the volume of URLs you thought you would have.<\/p>\n<p>Don\u2019t forget that having more than one language or location targeted at your domain may yield multiples of each page.<\/p>\n<p>Set your crawling tool to crawl as Googlebot or Bingbot and let it loose on all pages that these search bots would be able to access. This will give you a more accurate picture of the size of your website as they know it.<\/p>\n<h2 class=\"wp-block-heading\" id=\"h-why-crawl-budget-is-important\"><span class=\"ez-toc-section\" id=\"Why_crawl_budget_is_important\"><\/span>Why crawl budget is important<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Why is Google recommending that the above three types of sites consider their crawl budget? Why is it important to monitor and manage it?<\/p>\n<p>If your crawl budget is too low to allow the search bots to discover all the new URLs you\u2019ve added to your site or to revisit URLs that have changed, then they won\u2019t know about the content on them.<\/p>\n<p>That means the pages may not be indexed or if they are, they may not rank as well as they could if the bots could crawl them.<\/p>\n<h2 class=\"wp-block-heading\" id=\"h-how-crawl-budget-issues-happen\"><span class=\"ez-toc-section\" id=\"How_crawl_budget_issues_happen\"><\/span>How crawl budget issues happen<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Three main factors that can cause crawl budget issues:\u00a0<\/p>\n<ul class=\"wp-block-list\">\n<li>The quality of URLs.<\/li>\n<li>The volume of URLs.<\/li>\n<li>Their accessibility.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\" id=\"h-quality\"><span class=\"ez-toc-section\" id=\"Quality\"><\/span>Quality<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>We know that Google <span style=\"box-sizing: border-box; margin: 0px; padding: 0px;\">considers other pages\u00a0on a website when deciding whether to crawl new pages it has discovered<\/span>.\u00a0<\/p>\n<p>Googlebot may decide a page isn\u2019t worth the resources to crawl if it anticipates its content will not be of high enough value to index. This can be due to:<\/p>\n<ul class=\"wp-block-list\">\n<li>High volumes of on-site duplicate content.<\/li>\n<li>Hacked pages with poor-quality content.<\/li>\n<li>Internally created low-quality and spam content.\u00a0<\/li>\n<\/ul>\n<p>Poor-quality pages may have been intentionally created, either internally or by external bad actors. They may also be an unintended side effect of poor design and copy.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-volume\"><span class=\"ez-toc-section\" id=\"Volume\"><\/span>Volume<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Your site may have more URLs than you realize, often due to common technical issues like faceted navigation and infinite URL creation.<\/p>\n<p><strong>Faceted navigation<\/strong><\/p>\n<p>Faceted navigation is usually found on ecommerce websites.\u00a0<\/p>\n<p>If you have a category page like <code>www.example-pet-store.com\/cats\/toys<\/code>, you may have a filtering system to help users narrow down the products on that page.\u00a0<\/p>\n<p>If you want to narrow down the cat toy products in this fictitious pet store, you may select the \u201ccontains cat nip\u201d filter.\u00a0<\/p>\n<p>That may then yield a URL that looks something like this:\u00a0<\/p>\n<ul class=\"wp-block-list\">\n<li><code>www.example-pet-store.com\/cats\/toys?contains=catnip<\/code><\/li>\n<\/ul>\n<p>This is faceted navigation.<\/p>\n<p>Now, consider if the users want to narrow the search down even further to toys that have feathers.\u00a0<\/p>\n<p>They might end up on a URL like this one:\u00a0<\/p>\n<ul class=\"wp-block-list\">\n<li><code>www.example-pet-store.com\/cats\/toys?contains=catnip&amp;design=feathers\u00a0<\/code><\/li>\n<\/ul>\n<p>What about if they want to sort the list by price?\u00a0<\/p>\n<p>Clicking the sort button may take them to a new URL:\u00a0<\/p>\n<ul class=\"wp-block-list\">\n<li><code>www.example-pet-store.com\/cats\/toys?contains=catnip&amp;design=feathers&amp;sort=low<\/code><\/li>\n<\/ul>\n<p>You can see how quickly additional URLs are created stemming from one category page.\u00a0<\/p>\n<p>If Googlebot can find these pages, either through internal or external links, or perhaps they have been included in the XML sitemap, it may crawl them.<\/p>\n<p>Pretty soon, instead of crawling your site\u2019s 200 category pages and individual product pages, Googlebot might be aware of thousands of variants of the category pages.\u00a0<\/p>\n<p>As these filtering systems lead to new URLs being created, they can all be crawled unless you stop the bots from doing so or they deem the pages too low-value to do so.<\/p>\n<p><strong>Infinite URL creation<\/strong><\/p>\n<p>Events calendar. Book a table. Reserve a space.<\/p>\n<p>These types of date-based systems on websites that allow users to click through to future days or months can cause \u201cbot traps.\u201d<\/p>\n<p>Picture an events calendar. It shows the whole month with a highlight on the days with events.\u00a0<\/p>\n<p>It sits on the URL <code>\/events-calendar<\/code> and if you are looking at the month of January 2025, the URL will contain <code>\/events-calendar\/january-2025<\/code>. This is pretty common practice.<\/p>\n<p>If that calendar also has a button at the top that allows users to click through to the next month\u2019s events, that wouldn\u2019t be abnormal either.\u00a0<\/p>\n<p>Clicking once to view the next month\u2019s events might take you to a URL containing <code>\/events-calendar\/February<\/code>.\u00a0<\/p>\n<p>Click again, and you might end up on <code>\/events-calendar\/march-2025<\/code>.<\/p>\n<p>However, the real fun comes when there is no limit to how far into the future you can click.\u00a0<\/p>\n<p>Click on \u201cview next month\u2019s events\u201d enough times, and you could end up on <code>\/events-calendar\/december-2086<\/code>.<\/p>\n<p>If the calendar is set up in such a way that the \u201cview next month\u2019s events\u201d link changes on each page to be the next URL in the sequence of months, then the search bots could also end up following the links all the way through to <code>\/events-calendar\/december-2086<\/code> \u2013 and beyond.<\/p>\n<p>It\u2019s not useful content on page <code>\/events-calendar\/december-2086<\/code>. There probably haven\u2019t been any events organized yet.<\/p>\n<p>All of the resources wasted on those empty calendar pages could have been utilized by the bots on new products just uploaded to the site.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-accessibility\"><span class=\"ez-toc-section\" id=\"Accessibility\"><\/span>Accessibility<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Search bots may reduce the frequency of crawling a URL if it returns a server response code other than 200.\u00a0<\/p>\n<p>For example, a 4XX code indicates that the page cannot or should not be found, leading to less frequent crawling of that page.\u00a0<\/p>\n<p>Similarly, if multiple URLs return codes like 429 or 500, bots may reduce the crawling of those pages and eventually drop them from the index.<\/p>\n<p>Redirects can also impact crawling, albeit to a smaller extent. However, excessive use, such as long chains of redirects, can have a cumulative effect over time.<\/p>\n<p><!-- START INLINE FORM --><\/p>\n<p><!-- END INLINE FORM --><\/p>\n<hr class=\"wp-block-separator has-text-color has-cyan-bluish-gray-color has-css-opacity has-cyan-bluish-gray-background-color has-background\">\n<h2 class=\"wp-block-heading\" id=\"h-how-to-identify-crawl-budget-problems\"><span class=\"ez-toc-section\" id=\"How_to_identify_crawl_budget_problems\"><\/span>How to identify crawl budget problems<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It\u2019s impossible to determine if your site is suffering from crawl budget issues by looking at it alone.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-see-what-the-search-engines-are-reporting\"><span class=\"ez-toc-section\" id=\"See_what_the_search_engines_are_reporting\"><\/span>See what the search engines are reporting<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The first step to identifying if search bots are having issues crawling your site is to use their webmaster tools.\u00a0<\/p>\n<p>For example, look at the \u201cCrawl stats\u201d report in Google Search Console.\u00a0<\/p>\n<p>This will help you identify if a problem on your site may have caused Googlebot to increase or decrease its crawling.<\/p>\n<p>Also, have a look at the \u201cPage indexing\u201d report. Here, you will see the ratio between your site\u2019s indexed and unindexed pages.\u00a0<\/p>\n<p>When looking through the reasons for not indexing pages, you may also see crawl issues reported, such as \u201cDiscovered \u2013 currently not indexed.\u201d\u00a0<\/p>\n<p>This can be your first indication that pages on your site do not meet Google\u2019s crawling criteria.<\/p>\n<p><strong><em>Dig deeper: <\/em><\/strong><strong><em>Decoding Googlebot crawl stats data in Google Search Console<\/em><\/strong><\/p>\n<h3 class=\"wp-block-heading\" id=\"h-log-files\"><span class=\"ez-toc-section\" id=\"Log_files\"><\/span>Log files<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Another way to tell if the search bots are struggling to crawl your pages as much as they would like to is to analyze your log files.\u00a0<\/p>\n<p>Log files report any human users or bots that have \u201chit\u201d your website.<\/p>\n<p>By reviewing your site\u2019s log files, you can understand which pages have not been crawled by the search bots for a while.\u00a0<\/p>\n<p>If these are pages that are new or updated regularly, this can indicate that there may be a crawl budget problem.<\/p>\n<p><strong><em>Dig deeper. Crawl efficacy: How to level up crawl optimization<\/em><\/strong><\/p>\n<h2 class=\"wp-block-heading\" id=\"h-how-to-fix-crawl-budget-problems\"><span class=\"ez-toc-section\" id=\"How_to_fix_crawl_budget_problems\"><\/span>How to fix crawl budget problems<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Before trying to fix a crawl budget issue, ensure you have one.\u00a0<\/p>\n<p>Some of the fixes I\u2019m about to suggest are good practices for helping search bots focus on the pages you want them to crawl.\u00a0<\/p>\n<p>Others are more serious and could have a negative impact on your crawling if not applied carefully.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-another-word-of-warning\"><span class=\"ez-toc-section\" id=\"Another_word_of_warning\"><\/span>Another word of warning<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Carefully consider whether you\u2019re addressing a crawling or indexing issue before making changes.<\/p>\n<p>I\u2019ve seen many cases where pages are already in the index, and someone wants them removed, so they block crawling of those pages.<\/p>\n<p>This approach won\u2019t remove the pages from the index \u2013 at least not quickly.\u00a0<\/p>\n<p>Worse, they sometimes double down by adding a noindex meta tag to the pages they\u2019ve already blocked in the robots.txt file.<\/p>\n<p>The problem?\u00a0<\/p>\n<p>If crawling is blocked, search bots can\u2019t access the page to see the noindex tag, rendering the effort ineffective.<\/p>\n<p>To avoid such issues, don\u2019t mix crawling and indexing solutions.\u00a0<\/p>\n<p>Determine whether your primary concern is with crawling or indexing, and address that issue directly.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-fixing-crawl-budget-issues-through-the-robots-txt\"><span class=\"ez-toc-section\" id=\"Fixing_crawl_budget_issues_through_the_robotstxt\"><\/span>Fixing crawl budget issues through the robots.txt<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The robots.txt is a very valid way of helping the search bots determine which pages you do not want them crawling.\u00a0<\/p>\n<p>The \u201cdisallow\u201d command essentially prevents <strong>good<\/strong><em> <\/em>bots from crawling any URLs that match the disallow command.<\/p>\n<p><strong>Bad<\/strong><em> <\/em>bots can and do ignore the disallow command, so if you find your site is getting overwhelmed by bots of another nature, such as competitors scraping it, they may need to be blocked in another way.<\/p>\n<p>Check if your robots.txt file is blocking URLs that you want search bots to crawl. I\u2019ve used the <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/technicalseo.com\/tools\/robots-txt\/\" target=\"_blank\" rel=\"noopener\">robots.txt tester<\/a> from Dentsu to help with this.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-improving-the-quality-and-load-speed-of-pages\"><span class=\"ez-toc-section\" id=\"Improving_the_quality_and_load_speed_of_pages\"><\/span>Improving the quality and load speed of pages<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If search bots struggle to navigate your site, speeding up page loading can help.\u00a0<\/p>\n<p>Load speed is important for crawling, both the time it takes for the server to respond to a search bot\u2019s request and the time it takes to render a page.\u00a0<\/p>\n<p>Test the templates used on URLs that aren\u2019t being crawled regularly and see if they are slow-loading.<\/p>\n<p>Another reason you may not see pages being crawled, even for the first time, is because of quality.\u00a0<\/p>\n<p>Audit the pages not being crawled and those that perhaps share the same sub-folder but have been crawled.\u00a0<\/p>\n<p>Make sure that the content on those pages isn\u2019t too thin, duplicated elsewhere on the site or spammy.<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-control-crawling-through-robots-txt\"><span class=\"ez-toc-section\" id=\"Control_crawling_through_robotstxt\"><\/span>Control crawling through robots.txt<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>You can stop search bots from crawling single pages and entire folders through the robots.txt.\u00a0<\/p>\n<p>Using the \u201cdisallow\u201d command can help you decide which parts of your website you want bots to visit.<\/p>\n<p>For example, you may not want the search bots wasting crawl budget on your filtered category page results.\u00a0<\/p>\n<p>You could disallow the bots from crawling any page with the sorting or filtering parameters in the URL, like \u201c<code>?sort=<\/code>\u201d or \u201c<code>?content=<\/code>.\u201d<\/p>\n<h3 class=\"wp-block-heading\" id=\"h-consider-nofollow-links-on-internal-links\"><span class=\"ez-toc-section\" id=\"Consider_nofollow_links_on_internal_links\"><\/span>Consider nofollow links on internal links<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Another way to prevent bots from crawling certain pages is to add the \u201cnofollow\u201d attribute to the link tag.\u00a0<\/p>\n<p>With the events calendar example earlier, each \u201cView next month\u2019s events\u201d link could have the \u201cnofollow\u201d attribute. That way, human visitors could still click the link, but bots would not be able to follow it.<\/p>\n<p>Remember to add the \u201cnofollow\u201d attribute to the links wherever they appear on your site.\u00a0<\/p>\n<p>If you don\u2019t do this or someone adds a link to a deeper page in the events calendar system from their own site, the bots could still crawl that page.\u00a0\u00a0<\/p>\n<h2 class=\"wp-block-heading\" id=\"h-navigating-crawl-budget-for-seo-success-in-2025\"><span class=\"ez-toc-section\" id=\"Navigating_crawl_budget_for_SEO_success_in_2025\"><\/span>Navigating crawl budget for SEO success in 2025<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Most sites won\u2019t need to worry about their crawl budget or whether bots can access all the pages within the allocated time and resources.\u00a0<\/p>\n<p>However, that doesn\u2019t mean they should ignore how bots are crawling the site.\u00a0<\/p>\n<p>Even if you\u2019re not running out of crawl budget, there may still be issues preventing search bots from crawling certain pages, or you might be allowing them to crawl pages you don\u2019t want them to.<\/p>\n<p>It\u2019s important to monitor the crawling of your site as part of its overall technical health.\u00a0<\/p>\n<p>This way, if any issues arise that could hinder bots from crawling your content, you\u2019ll be aware and can address them promptly.<\/p>\n<p><strong><em>Dig deeper: <\/em><\/strong><strong><em>Top 6 technical SEO action items for 2025<\/em><\/strong><\/p>\n<\/div>\n<p><\/p>\n<div class=\"about-author\">\n<p>About the author<\/p>\n<div class=\"information\">\n<div class=\"author-module\">\n<div class=\"row\">\n<div class=\"col-12 col-lg-3 text-center\">\n<div class=\"avatar\">\n\t\t\t\t\t\t<img loading=\"lazy\" decoding=\"async\" class=\"img-fluid rounded-circle avatar-border\" alt=\"Helen Pollitt\" width=\"140\" height=\"140\" src=\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/12\/Helen-Pollitt-scaled.jpg.webp\"><img loading=\"lazy\" decoding=\"async\" class=\"img-fluid rounded-circle avatar-border\" src=\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/12\/Helen-Pollitt-scaled.jpg.webp\" alt=\"Helen Pollitt\" width=\"140\" height=\"140\">\n\t\t\t\t\t\t\t\t\t\t\t<\/div>\n<\/p><\/div>\n<div class=\"col-12 col-lg-9\">\n<div class=\"about\">\n<div class=\"name\">\n\t\t\t\t\t\t\t<strong>Helen Pollitt<\/strong>\n\t\t\t\t\t\t<\/div>\n<div class=\"row g-2 pt-2\">\n<div class=\"col-auto twitter\">\n\t\t\t\t\t\t\t\t\t<a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/twitter.com\/intent\/follow?original_referer=https%3A%2F%2Fsearchengineland.com%2F&amp;region=follow_link&amp;screen_name=HelenPollitt1&amp;tw_p=followbutton&amp;variant=2.0\" rel=\"me\" target=\"_blank\" aria-label=\"opens in a new tab\"><i class=\"fab fa-x-twitter\"><\/i><\/a>\n\t\t\t\t\t\t\t<\/div>\n<div class=\"col-auto\">\n\t\t\t\t\t\t\t\t\t<a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/www.linkedin.com\/in\/helen-p-b19500163\/\" target=\"_blank\" aria-label=\"opens in a new tab\"><i class=\"fab fa-linkedin\"><\/i><\/a>\n\t\t\t\t\t\t\t\t<\/div>\n<\/p><\/div>\n<p>\t\t\t\t\t\tHelen is a senior SEO with over a decade&#8217;s experience in the industry. She has a passion for equipping teams and training individuals in SEO strategy and tactics. Helen is often seen on stage at conferences delivering talks about digital marketing.\t\t\t\t\t<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/category\/technology\/\" target=\"_blank\" >Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/searchengineland.com\/crawl-budget-what-you-need-to-know-in-2025-448961\" target=\"_blank\" >Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Confused about crawl budget? This comprehensive guide explains it all \u2013 from server capacity to fixing crawling issues for better indexing. Crawl budget is a common source of concern and confusion in SEO.\u00a0 This guide will explain everything you need to know about crawl budget and how it may impact your technical SEO efforts in&#8230;<\/p>\n","protected":false},"author":1,"featured_media":646179,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/12\/Crawl-budget-What-you-need-to-know-in-2025.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[78070,148084],"class_list":["post-646178","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-seo","tag-technical-optimization"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/646178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=646178"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/646178\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/646179"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=646178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=646178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=646178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}