{"id":264721,"date":"2021-06-02T15:31:00","date_gmt":"2021-06-02T12:31:00","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-mount-an-s3-bucket-locally-on-linux-cloudsavvy-it\/"},"modified":"2021-06-02T15:31:00","modified_gmt":"2021-06-02T12:31:00","slug":"how-to-mount-an-s3-bucket-locally-on-linux-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-mount-an-s3-bucket-locally-on-linux-cloudsavvy-it\/","title":{"rendered":"#How to Mount an S3 Bucket Locally on Linux \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2910f683ac8\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2910f683ac8\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-mount-an-s3-bucket-locally-on-linux-cloudsavvy-it\/#The_Benefits_and_Limitations_of_S3_as_a_Filesystem\" >The Benefits and Limitations of S3 as a Filesystem<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-mount-an-s3-bucket-locally-on-linux-cloudsavvy-it\/#Setting_Up_s3fs-fuse\" >Setting Up s3fs-fuse<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Mount an S3 Bucket Locally on Linux \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage alignnone size-full wp-image-5269\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"AWS Logo\" width=\"700\" height=\"300\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>In many ways, S3 buckets act like like cloud hard drives, but are only \u201cobject level storage,\u201d not block level storage like EBS or EFS. However, it is possible to mount a bucket as a filesystem, and access it directly by reading and writing files.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"The_Benefits_and_Limitations_of_S3_as_a_Filesystem\"><\/span>The Benefits and Limitations of S3 as a Filesystem<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The magic that makes this whole setup work is a utility called <code>s3fs-fuse<\/code>. FUSE stands for Filesystem in Userspace, and it creates a mounted virtual filesystem. <code>s3fs<\/code>\u00a0interfaces with S3, and supports a large subset of POSIX, including reading, writing, creating directories, and setting file metadata.<\/p>\n<p>One of the great benefits of using S3 over traditional storage is that it\u2019s\u00a0<em>very<\/em> effective at storing individual objects long term, with no limit at all on total bucket size. You can store 10 photos or 10 million photos in S3, and it\u2019ll work largely the same. In <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>lications where you need a large (and cheap) disk, S3 makes sense, and if the application you\u2019re integrating wants file access, this is a good way to bridge the two.<\/p>\n<p>Of course, it\u2019s not without limitations. While it works fairly comparatively to an S3 API in terms of performance when storing and retrieving whole files, it obviously doesn\u2019t replace the much faster network attached block storage entirely. There\u2019s a reason this configuration isn\u2019t officially supported by AWS\u2014you\u2019ll run into concurrency issues with multiple clients using files, especially if you have clients in different regions accessing the same bucket. Of course, S3 also has this limitation, and it doesn\u2019t prevent you from having multiple clients attached, but it\u2019s more apparent when FUSE seems to give you \u201cdirect\u201d access. It\u2019s not, and you\u2019ll have to keep this limitations in mind.<\/p>\n<p>AWS does have a service similar to this\u2014<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/aws.amazon.com\/storagegateway\/?nc=sn&amp;loc=1&amp;tag=reviewgeek-20\">Storage Gateway<\/a>, which can act as a local NAS and provides local block storage backed by S3. However, this is more of an enterprise solution, and it requires an entire physical server to deploy a VMWare image to. <code>s3fs<\/code>, on the other hand, is a simple single server solution, although it doesn\u2019t do much caching.<\/p>\n<p>So, if you can convert applications to using the S3 API rather than a FUSE, you should do that instead. But, if you\u2019re okay with a bit of a hacky solution, <code>s3fs<\/code>\u00a0can be useful.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Setting_Up_s3fs-fuse\"><\/span>Setting Up s3fs-fuse<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Compared to how hacky it is, it\u2019s surprisingly easy to set up. <code>s3fs-fuse<\/code>\u00a0is available <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/github.com\/s3fs-fuse\/s3fs-fuse\">from most package managers<\/a>, though it may just be called <code>s3fs<\/code>\u00a0on some systems. For Debian-based systems like Ubuntu, that would be:<\/p>\n<pre>sudo apt install s3fs<\/pre>\n<p>You\u2019ll need to <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/console.aws.amazon.com\/iam\/home?tag=reviewgeek-20\">create an IAM user<\/a>, and give it permission to access the bucket you wish to mount. At the end, you\u2019ll get a secret access key:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"imgchk9 alignnone wp-image-5101 size-full\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/6ce9cad8.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Successfully setting up s3fs-fuse and receiving a secret access key.\" width=\"700\" height=\"261\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>You can paste these in the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/cli-configure-files.html?tag=reviewgeek-20\">standard AWS credentials file<\/a>, <code>~\/.aws\/credentials<\/code>, but if you want to use a different key, <code>s3fs<\/code>\u00a0supports a custom password file.\u00a0Paste both the access key ID and secret into <code>\/etc\/passwd-s3fs<\/code> , in the following format:<\/p>\n<pre>echo ACCESS_KEY_ID:SECRET_ACCESS_KEY &gt; \/etc\/passwd-s3fs<\/pre>\n<p>And make sure the permissions on this keyfile are set properly, or it\u2019ll complain:<\/p>\n<pre>chmod 600 \/etc\/passwd-s3fs<\/pre>\n<p>Then, you can mount the bucket with the following command:<\/p>\n<pre>s3fs bucket-name \/mnt\/bucket-name<\/pre>\n<p>If that doesn\u2019t work, you can enable debug output with a few extra flags:<\/p>\n<pre>-o dbglevel=info -f -o curldbg<\/pre>\n<p>If you want this to mount at boot, you\u2019ll need to add the following to your <code>\/etc\/fstab<\/code>:<\/p>\n<pre>s3fs#bucket-name \/mnt\/bucket-name fuse _netdev,allow_other,umask=227,uid=33,gid=33,use_cache=\/root\/cache 0 0<\/pre>\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/5099\/how-to-mount-an-s3-bucket-locally-on-linux\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Mount an S3 Bucket Locally on Linux \u2013 CloudSavvy IT&#8221; In many ways, S3 buckets act like like cloud hard drives, but are only \u201cobject level storage,\u201d not block level storage like EBS or EFS. However, it is possible to mount a bucket as a filesystem, and access it directly by reading and&#8230;<\/p>\n","protected":false},"author":1,"featured_media":264722,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-264721","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/264721","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=264721"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/264721\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/264722"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=264721"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=264721"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=264721"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}