{"id":135630,"date":"2020-12-17T16:00:00","date_gmt":"2020-12-17T13:00:00","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-rotate-and-delete-old-elasticsearch-records-after-a-month-cloudsavvy-it\/"},"modified":"2020-12-17T16:00:00","modified_gmt":"2020-12-17T13:00:00","slug":"how-to-rotate-and-delete-old-elasticsearch-records-after-a-month-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-rotate-and-delete-old-elasticsearch-records-after-a-month-cloudsavvy-it\/","title":{"rendered":"#How To Rotate and Delete Old Elasticsearch Records After a Month \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a3ae90a96e2e\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a3ae90a96e2e\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-rotate-and-delete-old-elasticsearch-records-after-a-month-cloudsavvy-it\/#Deleting_Using_The_%E2%80%9CDelete_By_Query%E2%80%9D_API\" >Deleting Using The \u201cDelete By Query\u201d API<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-rotate-and-delete-old-elasticsearch-records-after-a-month-cloudsavvy-it\/#A_Better_Method_Time_Based_Indices\" >A Better Method: Time Based Indices<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How To Rotate and Delete Old Elasticsearch Records After a Month \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-8620\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/a21db152f1800f6598017f8ef8f95d6a\/p\/uploads\/2020\/12\/4c81ae38.png\" alt=\"\" width=\"700\" height=\"299\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Elasticsearch indices can quickly fill up with gigabytes of data, especially if you\u2019re logging from multiple servers many times a second. To manage data, Elasticsearch<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Deleting_Using_The_%E2%80%9CDelete_By_Query%E2%80%9D_API\"><\/span>Deleting Using The \u201cDelete By Query\u201d API<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Elasticsearch offers a \u201cDelete By Query\u201d API, that will remove all documents matching a query. You can use this to match timestamps greater or less than a certain date, albeit a bit crudely:<\/p>\n<pre>POST indexname\/_delete_by_query&#13;\n{&#13;\n  \"query\": {&#13;\n    \"range\" : {&#13;\n      \"@timestamp\" : {&#13;\n        \"gte\" : \"09\/02\/2020\",&#13;\n        \"lte\" : \"11\/02\/2020\",&#13;\n        \"format\": \"dd\/MM\/yyyy||yyyy\"&#13;\n      }&#13;\n    }&#13;\n  }&#13;\n}<\/pre>\n<p>However, this query is\u00a0<strong><em>really slow<\/em><\/strong>. It scales linearly with document size. If you have enough documents that you need to be rotating them to prevent your Elasticsearch instance from bursting into flames, you probably can\u2019t delete records this way, and will need to use time-based indices instead.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"A_Better_Method_Time_Based_Indices\"><\/span>A Better Method: Time Based Indices<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In Elasticsearch, you don\u2019t usually use indexes directly. Your dashboards use index patterns, which can match multiple indexes at once. The reason for this is that the indexes themselves can act as groups of data, such as grouping by day or month.<\/p>\n<p>It\u2019s much easier to manage and rotate entire indices, so if you had each ingester configured to add the current date to the index name,<\/p>\n<pre>index: \"indexname-%{+yyyy.MM.dd}\"<\/pre>\n<p>Of course, this requires you to configure the ingest pipeline to write to the daily index. You\u2019ll need to set up your loggers to ingest data in this format.<\/p>\n<p>Once that\u2019s done though, you can create a new Index Lifecycle Policy to handle the automatic rollover of data. This option is available under \u201cStack Management\u201d in the Kibana dashboard.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-8621\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/6d473dab709d9b88357438ae9d4514e7\/p\/uploads\/2020\/12\/00a665be.png\" alt=\"\" width=\"700\" height=\"252\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>You can configure multiple phases of index rollover, but for this purpose it\u2019s easier to just disable rollover and enable the delete phase, configuring it to remove indices older than X number of days.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-8622\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/b5117be6790b9b00c513f147274c4565\/p\/uploads\/2020\/12\/745371ea.png\" alt=\"\" width=\"700\" height=\"474\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Then, to actually <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ly it to an index template, you\u2019ll need to select \u201cAdd Policy To Index Template\u201d under \u201cActions\u201d in the lifecycle policy list.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-8623\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/aa17a62675ad28812dea9f11593ec347\/p\/uploads\/2020\/12\/fc6f61ff.png\" alt=\"\" width=\"421\" height=\"182\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Select the index pattern you wish to add, and the policy should take effect im<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a>tely, and your old indices in the pattern will be deleted.\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/7152\/how-to-rotate-and-delete-old-elasticsearch-records-after-a-month\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How To Rotate and Delete Old Elasticsearch Records After a Month \u2013 CloudSavvy IT&#8221; Elasticsearch indices can quickly fill up with gigabytes of data, especially if you\u2019re logging from multiple servers many times a second. To manage data, Elasticsearch Deleting Using The \u201cDelete By Query\u201d API Elasticsearch offers a \u201cDelete By Query\u201d API, that will&#8230;<\/p>\n","protected":false},"author":1,"featured_media":135631,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/12\/4c81ae38.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-135630","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/135630","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=135630"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/135630\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/135631"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=135630"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=135630"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=135630"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}