{"id":109160,"date":"2020-11-10T17:00:16","date_gmt":"2020-11-10T14:00:16","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-get-the-largest-items-in-an-s3-bucket-cloudsavvy-it\/"},"modified":"2020-11-10T17:00:16","modified_gmt":"2020-11-10T14:00:16","slug":"how-to-get-the-largest-items-in-an-s3-bucket-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-get-the-largest-items-in-an-s3-bucket-cloudsavvy-it\/","title":{"rendered":"#How To Get The Largest Items In An S3 Bucket \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2a053760bb1\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2a053760bb1\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-get-the-largest-items-in-an-s3-bucket-cloudsavvy-it\/#Listing_and_Sorting_Items_with_the_S3_CLI\" >Listing and Sorting Items with the S3 CLI<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How To Get The Largest Items In An S3 Bucket \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-5269\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/0eb3564906a864c93706b30eaca199af\/p\/uploads\/2020\/06\/e601b806.png\" alt=\"AWS Logo\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>If you\u2019re curious about the largest items in an AWS S3 bucket, you can use the CLI to print out a list sorted by size. This can help you locate unusually large objects in the bucket which may be taking up space.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Listing_and_Sorting_Items_with_the_S3_CLI\"><\/span>Listing and Sorting Items with the S3 CLI<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>S3 provides some built in sorting options in the menus, so if you\u2019re just looking for the largest item in a folder, you can simply sort that folder. However, if you want to search for all items regardless of key, you\u2019ll need to do so from the AWS CLI. If you don\u2019t have that installed, you can refer to our guide on configuring it to set it up.<\/p>\n<p>The command for listing objects is pretty simple:<\/p>\n<pre>aws s3api list-object-versions --bucket example-bucket<\/pre>\n<p>This query can take a while to evaluate, since it\u2019s <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">download<\/a>ing a master list of all objects and their metadata, but you\u2019ll get a JSON array containing an entry for each item, like the following:<\/p>\n<pre>        {&#13;\n            \"ETag\": \"\"04e28fbee1ef2721123bb4e9a78183a895\"\",&#13;\n            \"Size\": 320,&#13;\n            \"StorageClass\": \"STANDARD\",&#13;\n            \"Key\": \"folder\/file.json\",&#13;\n            \"VersionId\": \"fNdwjJRaEjBYUSBgZe51oj_s4ONo5GsL\",&#13;\n            \"IsLatest\": false,&#13;\n            \"LastModified\": \"2020-11-05T18:59:18+00:00\",&#13;\n            \"Owner\": {&#13;\n                \"DisplayName\": \"username\",&#13;\n                \"ID\": \"501092a155f88f4d174d7as3d2a347f33b9495f0261434682ab9a\"&#13;\n            }&#13;\n        }<\/pre>\n<p>To parse and sort this, you can use <code>jq<\/code>\u00a0\u2014 a fantastic utility for working with JSON on the command line. You can download it from <code>apt<\/code>, though <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/stedolan.github.io\/jq\/download\/\">binaries are also available<\/a>:<\/p>\n<pre>sudo apt-get install jq<\/pre>\n<p>This makes the final command the following, which will still take a while to evaluate, but will print out the largest 100 items in the bucket:<\/p>\n<pre>aws s3api list-object-versions --bucket oxide.rust | jq -r '.Versions[] | \"(.Key)t (.Size)\"' | sort -k2 -r -n | head -100<\/pre>\n<p>If you want more or less items, you can change the input parameter to the <code>head<\/code>\u00a0command, which trims all but the first N lines.\n<\/div>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener noreferrer\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/7894\/how-to-get-the-largest-items-in-an-s3-bucket\/\" target=\"_blank\" rel=\"noopener noreferrer\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How To Get The Largest Items In An S3 Bucket \u2013 CloudSavvy IT&#8221; If you\u2019re curious about the largest items in an AWS S3 bucket, you can use the CLI to print out a list sorted by size. This can help you locate unusually large objects in the bucket which may be taking up space&#8230;.<\/p>\n","protected":false},"author":1,"featured_media":109161,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-109160","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/109160","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=109160"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/109160\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/109161"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=109160"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=109160"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=109160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}