{"id":258565,"date":"2021-05-25T19:00:00","date_gmt":"2021-05-25T16:00:00","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/"},"modified":"2021-05-25T19:00:00","modified_gmt":"2021-05-25T16:00:00","slug":"how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/","title":{"rendered":"#How to Clean Up Old Containers and Images in Your Kubernetes Cluster \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a409abda49bd\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a409abda49bd\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/#Container_Images\" >Container Images<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/#Clearing_Old_Containers\" >Clearing Old Containers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/#Should_I_Manually_Intervene\" >Should I Manually Intervene?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/#The_Future_Evictions\" >The Future: Evictions<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster-cloudsavvy-it\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Clean Up Old Containers and Images in Your Kubernetes Cluster \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage aligncenter size-full wp-image-9632\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/02\/748108a6.jpg?width=1200&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Graphic showing the Kubernetes logo\" width=\"1602\" height=\"902\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>An active Kubernetes cluster can accumulate old containers and images. Ensuring discarded resources are removed when redundant helps to free up resources on your cluster\u2019s nodes. Here\u2019s how to <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>roach garbage collection in Kubernetes.<\/p>\n<h2 id=\"container-images\"><span class=\"ez-toc-section\" id=\"Container_Images\"><\/span>Container Images<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Kubernetes has a built-in garabage collection system that can clean up unused images. It\u2019s managed by Kubelet, the Kubernetes worker process that runs on each node.<\/p>\n<p>Kubelet automatically monitors unused images and <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/kubelet-garbage-collection\">will remove them<\/a> periodically. Deletion decisions are made by assessing the image\u2019s disk usage and the time at which it was last used. A large image that has been unused for a week will usually be cleaned up before a small one that was used yesterday.<\/p>\n<p>You can customise when garbage collection runs by specifying high and low thresholds for disk usage. Disk usage above the \u201chigh\u201d threshold will trigger garbage collection. The procedure will try to reduce disk usage down to the \u201clow\u201d threshold.<\/p>\n<p>The thresholds are defined using two Kubelet flags:<\/p>\n<ul>\n<li><strong><code>image-gc-high-threshold<\/code><\/strong> \u2013 Sets the high threshold; defaults to 85%.<\/li>\n<li><strong><code>image-gc-low-threshold<\/code><\/strong> \u2013 Sets the low threshold; defaults to 80%.<\/li>\n<\/ul>\n<p>These settings should already be active in your cluster. Kubelet will try to bring disk usage down to 80% after it becomes 85% full.<\/p>\n<p>You can set Kubectl flags in <code>\/var\/lib\/kubelet\/kubeadm-flags.env<\/code>:<\/p>\n<pre>KUBELET_KUBEADM_ARGS=\"--image-gc-high-threshold=60 --image-gc-low-threshold=50\"<\/pre>\n<p>After editing the file, restart Kubectl:<\/p>\n<pre>systemctl daemon-reload&#13;\nsystemctl restart kubelet<\/pre>\n<h2 id=\"clearing-old-containers\"><span class=\"ez-toc-section\" id=\"Clearing_Old_Containers\"><\/span>Clearing Old Containers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Kubelet also handles clean up of redundant containers. Any containers which are stopped or unidentified will be candidates for removal.<\/p>\n<p>You can grant old containers a grace period before deletion by defining a minimum container age. Additional flags let you control the total number of dead containers allowed to exist in a single pod and on the node:<\/p>\n<ul>\n<li><strong><code>maximum-dead-containers<\/code><\/strong> \u2013 Maximum number of old containers to retain. When set to <code>-1<\/code> (the default), no limit applies.<\/li>\n<li><strong><code>maximum-dead-containers-per-container<\/code><\/strong> \u2013 Set the number of older instances to be retained on a per-container basis. If a container is replaced with a newer instance, this many older versions will be allowed to remain.<\/li>\n<li><strong><code>minimum-container-ttl-duration<\/code><\/strong> \u2013 Garbage collection grace period for dead containers. Once a container is this many minutes old, it becomes eligible for garbage collection. The default value of <code>0<\/code> means no grace period applies.<\/li>\n<\/ul>\n<p>You can configure these settings with Kubelet flags using the same procedure as described above.<\/p>\n<h2 id=\"should-i-manually-intervene\"><span class=\"ez-toc-section\" id=\"Should_I_Manually_Intervene\"><\/span>Should I Manually Intervene?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You should not make manual efforts to remove dead containers or images. If disk space is filling up, or garbage collection doesn\u2019t seem to be working, try adjusting your Kubelet flags towards more aggressive settings.<\/p>\n<p>Kubernetes <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/kubelet-garbage-collection\">warns against<\/a> performing external garbage collection. Don\u2019t manually delete resources, either using cluster management APIs or third-party tools. This risks creating an inconsistent state which could impact Kubelet\u2019s operation.<\/p>\n<p>Kubelet is responsible for managing the containers allocated to each node. When a new container gets scheduled, Kubelet will download its image. Successful cluster operation is dependent on Kubelet\u2019s expectations being met. A missing image or container can lead to Kubelet issues.<\/p>\n<h2 id=\"the-future-evictions\"><span class=\"ez-toc-section\" id=\"The_Future_Evictions\"><\/span>The Future: Evictions<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The settings described above are supported in current Kubernetes versions. However, they are being deprecated in favour of a more robust \u201cevictions\u201d system. Evictions are a unified way to clean up Kubernetes resources; they\u2019ll eventually replace garbage collection.<\/p>\n<p>An eviction can occur for several reasons. Kubelet will monitor multiple factors, including available hardware resources and user configuration for retention periods.<\/p>\n<p>This new system facilitates the removal of garbage collection as a dedicated mechanism. The same process which terminates pods due to a low-memory scenario will delete redundant images as disk space becomes constrained.<\/p>\n<p>Two types of eviction are defined: <em>hard<\/em> and <em>soft<\/em>. A hard eviction will take im<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a>te action to remove the target resource. There is no grace period. A soft eviction has a user-configured grace period; the resource will be targeted once the grace period expires. If the cause of the eviction gets resolved during the grace period, such as more disk space becoming available, the removal can be terminated.<\/p>\n<p>The evictions system <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/kubelet-garbage-collection\/#deprecation\">isn\u2019t yet<\/a> fully supported for container clean ups. The <code>dead-containers<\/code> flags are already deprecated, ready for the future. You can <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/kubernetes.io\/docs\/tasks\/administer-cluster\/out-of-resource\/#deprecation-of-existing-feature-flags-to-reclaim-disk\">aleady use it<\/a> with container images \u2013 set <code>--eviction-hard<\/code> or <code>--eviction-soft<\/code> instead of the <code>threshold<\/code> flags.<\/p>\n<pre>--eviction-hard=imagefs.available&lt;1Gi<\/pre>\n<p>This example instructs Kubelet to remove all unused container images if the available disk space for image storage drops below 1GB.<\/p>\n<pre>--eviction-soft=imagefs.available&lt;1Gi&#13;\n--eviction-soft-grace-period=imagefs.available=5m<\/pre>\n<p>This second example shows how a \u201csoft\u201d eviction can be used instead. In this case, images won\u2019t be deleted unless the available disk space has been below 1GB for at least five minutes.<\/p>\n<h2 id=\"summary\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Kubernetes has garbage collection enabled by default. Dead containers and redundant images will be cleaned up periodically. The default schedule targets disk usage of 80% or lower; containers are cleaned up quite aggressively once they\u2019ve been stopped. You can use Kubelet flags to adjust the thresholds in the process.<\/p>\n<p>Garbage collection as a concept will eventually be removed in favour of evictions. Evictions have a simplified configuration which better aligns with other forms of resource removal. You can setup evictions in your cluster today. Remember that the garbage collection flags do not map directly to their eviction counterparts.\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/11041\/how-to-clean-up-old-containers-and-images-in-your-kubernetes-cluster\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Clean Up Old Containers and Images in Your Kubernetes Cluster \u2013 CloudSavvy IT&#8221; An active Kubernetes cluster can accumulate old containers and images. Ensuring discarded resources are removed when redundant helps to free up resources on your cluster\u2019s nodes. Here\u2019s how to approach garbage collection in Kubernetes. Container Images Kubernetes has a built-in&#8230;<\/p>\n","protected":false},"author":1,"featured_media":258566,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/02\/748108a6.jpg","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-258565","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/258565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=258565"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/258565\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/258566"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=258565"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=258565"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=258565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}