{"id":340502,"date":"2021-09-17T12:00:38","date_gmt":"2021-09-17T09:00:38","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/"},"modified":"2021-09-17T12:00:38","modified_gmt":"2021-09-17T09:00:38","slug":"how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/","title":{"rendered":"#How to Set an Expiration Policy on DigitalOcean Spaces Buckets \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a3db48b8eed2\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a3db48b8eed2\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/#Getting_Started\" >Getting Started<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/#Creating_Your_Policy\" >Creating Your Policy<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/#Applying_the_Policy\" >Applying the Policy<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/#Using_Multiple_Rules\" >Using Multiple Rules<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/#Aborting_Failed_Uploads\" >Aborting Failed Uploads<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets-cloudsavvy-it\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Set an Expiration Policy on DigitalOcean Spaces Buckets \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage aligncenter size-full wp-image-8934\" srcset=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/01\/83680385.jpeg?width=398&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 400w, https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/01\/83680385.jpeg?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 1200w\" sizes=\"auto, 400w, 1200w\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/01\/83680385.jpeg?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"DigitalOcean logo\" width=\"1600\" height=\"900\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>The DigitalOcean Spaces web interface doesn\u2019t provide a way to set bucket lifecycle policies. Here\u2019s how to <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ly expiration settings that automatically delete files after a set time period. This makes Spaces a more suitable location for rotated backups and log files.<\/p>\n<p>Although this feature isn\u2019t in DigitalOcean\u2019s UI, it is supported by the Spaces backend. Spaces is compatible with Amazon S3 APIs so command-line clients can set S3-based lifecycle policies. The steps detailed below should also work with other object storage providers that implement S3 APIs.<\/p>\n<h2 id=\"getting-started\"><span class=\"ez-toc-section\" id=\"Getting_Started\"><\/span>Getting Started<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You\u2019ll need the AWS CLI installed to follow along with this tutorial. Once it\u2019s installed, the first task is to supply credentials so the CLI can access your DigitalOcean account.<\/p>\n<p>Head to the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/cloud.digitalocean.com\">DigitalOcean Control Panel<\/a> in your browser. Click the \u201cAPI\u201d link at the bottom of the blue sidebar to the left of your screen. Next, click the \u201cGenerate New Key\u201d button to the right of the \u201cSpaces access keys\u201d heading.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-14163\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/09\/2114ada3.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"1262\" height=\"591\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Give your new key a name, then click the checkmark to complete the process. Your key and its corresponding secret will be displayed. Take note of these values as it\u2019s impossible to retrieve the secret part after you leave the screen.<\/p>\n<p>Return to your terminal and run <code>aws configure<\/code>. You\u2019ll be asked for your access key and secret. Follow the interactive prompts to supply the values you generated in DigitalOcean\u2019s web interface.<\/p>\n<p>Unfortunately this still isn\u2019t the end of the CLI setup. A significant limitation of the official S3 client is its inability to save custom endpoint URLs alongside your credentials. This means you must explicitly specify the DigitalOcean API URL with every command you issue:<\/p>\n<pre>aws s3 ls --endpoint=https:\/\/nyc3.digitaloceanspaces.com --bucket my-bucket<\/pre>\n<p>The command above will display the objects in the <code>my-bucket<\/code> bucket of your Spaces account. If you omitted the <code>--endpoint<\/code> flag, the S3 CLI would assume you\u2019re trying to connect to an AWS account. The endpoint URL needs to match the DigitalOcean datacentre region you created your space in \u2013 substitute the <code>nyc3<\/code> subdomain for the region you\u2019re using.<\/p>\n<h2 id=\"creating-your-policy\"><span class=\"ez-toc-section\" id=\"Creating_Your_Policy\"><\/span>Creating Your Policy<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Bucket lifecycle policies are define as JSON files that describe the rules you want to apply. Create a new file using your favorite text editor and add the following content:<\/p>\n<div class=\"wp-geshi-highlight-wrap5\">\n<div class=\"wp-geshi-highlight-wrap4\">\n<div class=\"wp-geshi-highlight-wrap3\">\n<div class=\"wp-geshi-highlight-wrap2\">\n<div class=\"wp-geshi-highlight-wrap\">\n<div class=\"wp-geshi-highlight\">\n<div class=\"json\">\n<pre class=\"de1\">{&#13;\n    \"Rules\": [&#13;\n        {&#13;\n            \"ID\": \"Prune old files\",&#13;\n            \"Status\": \"Enabled\",&#13;\n            \"Prefix\": \"\",&#13;\n            \"Expiration\": {&#13;\n                \"Days\": 30&#13;\n            }&#13;\n        }&#13;\n    ]&#13;\n}<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>The JSON is a <em>declarative<\/em> representation of the policy to apply. The policy\u2019s attributes and its current state are both specified inside the file.<\/p>\n<p>This example will delete files 30 days after they\u2019re uploaded. Setting <code>Status<\/code> to <code>Enabled<\/code> activates the policy, while an empty <code>Prefix<\/code> applies it to every item in the bucket. You can use the <code>Prefix<\/code> field to selectively delete only certain objects, such as those in the <code>temp\/<\/code> subdirectory.<\/p>\n<h2 id=\"applying-the-policy\"><span class=\"ez-toc-section\" id=\"Applying_the_Policy\"><\/span>Applying the Policy<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Next you need to use the AWS CLI to apply your policy to your bucket:<\/p>\n<pre>aws s3api put-bucket-lifecycle-configuration &#13;\n    --bucket my-bucket&#13;\n    --endpoint https:\/\/nyc3.digitaloceanspaces.com&#13;\n    --lifecycle-configuration file:\/\/my-policy.json<\/pre>\n<p>Substitute <code>my-bucket<\/code> with the name of the bucket you want to use your expiration rules with.<\/p>\n<p>The CLI will read your policy JSON file and attach it to the bucket. As long as the <code>Status<\/code> is <code>Enabled<\/code>, the lifecycle rules will apply im<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a>tely. You\u2019ll start seeing newly uploaded objects leave your bucket as they pass the threshold for expiration.<\/p>\n<p>You can check your policy\u2019s been applied by reading it back from the CLI:<\/p>\n<pre>aws s3api get-bucket-lifecycle-configuration &#13;\n    --bucket my-bucket&#13;\n    --endpoint https:\/\/nyc3.digitaloceanspaces.com<\/pre>\n<p>This should show you the JSON you submitted.<\/p>\n<h2 id=\"using-multiple-rules\"><span class=\"ez-toc-section\" id=\"Using_Multiple_Rules\"><\/span>Using Multiple Rules<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You can include multiple items in your <code>Rules<\/code> JSON array. This lets you apply unique expiration policies to different groups of objects, using the <code>Prefix<\/code> field:<\/p>\n<div class=\"wp-geshi-highlight-wrap5\">\n<div class=\"wp-geshi-highlight-wrap4\">\n<div class=\"wp-geshi-highlight-wrap3\">\n<div class=\"wp-geshi-highlight-wrap2\">\n<div class=\"wp-geshi-highlight-wrap\">\n<div class=\"wp-geshi-highlight\">\n<div class=\"json\">\n<pre class=\"de1\">{&#13;\n    \"Rules\": [&#13;\n        {&#13;\n            \"ID\": \"Prune Invoices\",&#13;\n            \"Status\": \"invoice\",&#13;\n            \"Prefix\": \"I\",&#13;\n            \"Expiration\": {&#13;\n                \"Days\": 90&#13;\n            }&#13;\n        },&#13;\n        {&#13;\n            \"ID\": \"Prune Quotations\",&#13;\n            \"Status\": \"Enabled\",&#13;\n            \"Prefix\": \"Q\",&#13;\n            \"Expiration\": {&#13;\n                \"Days\": 30&#13;\n            }&#13;\n        }&#13;\n    ]&#13;\n}<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>This policy would delete quotations after 30 days while letting invoices live for 90 days. Each bucket supports up to 100 individual lifecycle rules.<\/p>\n<h2 id=\"aborting-failed-uploads\"><span class=\"ez-toc-section\" id=\"Aborting_Failed_Uploads\"><\/span>Aborting Failed Uploads<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Another role of lifecycle policies is cleaning up after failed multipart uploads. When you add large files via the S3 APIs, they\u2019re chunked into streamable sections to improve performance and resiliency to network dropouts.<\/p>\n<p>You can end up with partial chunks sitting in your bucket if an upload part fails to complete. Add the <code>AbortIncompleteMultipartUpload<\/code> field to a lifecycle policy to remove these redundant chunks.<\/p>\n<div class=\"wp-geshi-highlight-wrap5\">\n<div class=\"wp-geshi-highlight-wrap4\">\n<div class=\"wp-geshi-highlight-wrap3\">\n<div class=\"wp-geshi-highlight-wrap2\">\n<div class=\"wp-geshi-highlight-wrap\">\n<div class=\"wp-geshi-highlight\">\n<div class=\"json\">\n<pre class=\"de1\">{&#13;\n    \"Rules\": [&#13;\n        {&#13;\n            \"ID\": \"AbortIncompleteMultipartUpload\",&#13;\n            \"Prefix\": \"\",&#13;\n            \"Status\": \"Enabled\",&#13;\n            \"AbortIncompleteMultipartUpload\": {&#13;\n                \"DaysAfterInitiation\": 1&#13;\n            }&#13;\n        }&#13;\n    ]&#13;\n}<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>This policy cleans incomplete upload chunks one day after they were started, potentially freeing up some storage space. When chunks are deleted, you won\u2019t be able to resume the original upload again \u2013 clients will need to restart it from the beginning.<\/p>\n<h2 id=\"summary\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>DigitalOcean Spaces supports S3 lifecycle policies but you have to apply them using the API. Once configured, your uploads will be deleted automatically after a set time period, giving you confidence that old files aren\u2019t wasting storage space and raising your bill.<\/p>\n<p>Although Spaces implements expiration policies in the same way as S3, other forms of lifecycle policy aren\u2019t available on DigitalOcean\u2019s platform. Another key component of S3\u2019s featureset is the ability to transition objects between storage classes, such as an automatic migration to archival storage after 30 days. Billing for Spaces is much simpler with only one plan available so these policies won\u2019t have an effect on DigitalOcean buckets.\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/14162\/how-to-set-an-expiration-policy-on-digitalocean-spaces-buckets\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Set an Expiration Policy on DigitalOcean Spaces Buckets \u2013 CloudSavvy IT&#8221; The DigitalOcean Spaces web interface doesn\u2019t provide a way to set bucket lifecycle policies. Here\u2019s how to apply expiration settings that automatically delete files after a set time period. This makes Spaces a more suitable location for rotated backups and log files&#8230;.<\/p>\n","protected":false},"author":1,"featured_media":340503,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/01\/83680385.jpeg","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-340502","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/340502","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=340502"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/340502\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/340503"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=340502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=340502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=340502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}