{"id":293556,"date":"2021-07-08T04:50:00","date_gmt":"2021-07-08T01:50:00","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it\/"},"modified":"2021-07-08T04:50:00","modified_gmt":"2021-07-08T01:50:00","slug":"how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it\/","title":{"rendered":"#How to Sync Files from Linux to Amazon S3 \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2ef25910f9b\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2ef25910f9b\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it\/#Limit_S3_Access_to_an_IAM_User\" >Limit S3 Access to an IAM User<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it\/#File_Syncing_With_s3cmd\" >File Syncing With s3cmd<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it\/#Another_Option_AWS_CLI\" >Another Option: AWS CLI<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-sync-files-from-linux-to-amazon-s3-cloudsavvy-it\/#Full_Backups_Restic_Duplicity\" >Full Backups: Restic,\u00a0Duplicity<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Sync Files from Linux to Amazon S3 \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage  alignnone wp-image-596 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2019\/06\/55634f08.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Amazon S3.\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>AWS S3 is Amazon\u2019s cloud storage service, allowing you to store individual files as objects in a bucket. You can upload files from the command line on your Linux server, or even sync entire directories to S3.<\/p>\n<p>If you just want to share files between EC2 instances, you can use an <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/aws.amazon.com\/efs\/?tag=reviewgeek-20\">EFS volume<\/a>\u00a0and mount it directly to multiple servers, cutting out the \u201ccloud\u201d altogether. But you shouldn\u2019t use it for everything, because it\u2019s <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/aws.amazon.com\/efs\/pricing\/?tag=reviewgeek-20\">much pricier<\/a> than S3, even with Infrequent Access turned on.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Limit_S3_Access_to_an_IAM_User\"><\/span>Limit S3 Access to an IAM User<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Your server probably doesn\u2019t need full root access to your AWS account, so before you do any kind of file syncing, you should make a new IAM user for your server to use. With an IAM user, you can limit your server to only managing your S3 buckets.<\/p>\n<p>From the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/console.aws.amazon.com\/iam\/home?tag=reviewgeek-20\">IAM Management Console<\/a>, make a new user, and enable \u201cProgrammatic Access.\u201d<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" alignnone wp-image-579 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2019\/06\/277b1aca-3.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Set User Details menu.\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>You\u2019ll be asked to choose permissions for this user. Make a new group, and assign it the \u201cAmazonS3FullAccess\u201d permission.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" alignnone wp-image-580 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2019\/06\/cb42aca9-1.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Assigning group permissions.\" width=\"700\" height=\"287\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>After that, you\u2019ll be given an access key and secret key. Make a note of these; you\u2019ll need them to authenticate your server.<\/p>\n<p>You can also manually assign more detailed S3 permissions, such as permission to use a specific bucket or only to upload files, but limiting access to just S3 should be fine in most cases.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"File_Syncing_With_s3cmd\"><\/span>File Syncing With s3cmd<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><code>s3cmd<\/code>\u00a0is a utility designed to make working with S3 from the command line easier. It\u2019s not a part of the AWS CLI, so you\u2019ll have to manually install it from your distro\u2019s package manager. For Debian-based systems like Ubuntu, that would be:<\/p>\n<pre>sudo apt-get install s3cmd<\/pre>\n<p>Once <code>s3cmd<\/code> is installed, you\u2019ll need to link it to the IAM user you created to manage S3. Run the configuration with:<\/p>\n<pre>s3cmd --configure<\/pre>\n<p>You\u2019ll be asked for the access key and secret key that the IAM Management Console gave you. Paste those in here. There\u2019s a few more options, such as changing the endpoints for S3 or enabling encryption, but you can leave them all default and just select \u201cY\u201d at the end to save the configuration.<\/p>\n<p>To upload a file, use:<\/p>\n<pre>s3cmd put file s3:\/\/bucket<\/pre>\n<p>Replacing \u201cbucket\u201d with your bucket name. To retrieve those files, run:<\/p>\n<pre>s3cmd get s3:\/\/bucket\/remotefile localfile<\/pre>\n<p>And, if you want to sync over a whole directory, run:<\/p>\n<pre>s3cmd sync directory s3:\/\/bucket\/<\/pre>\n<p>This will copy the entire directory into a folder in S3. The next time you run it, it will only copy the files that have changed since it was last ran. It won\u2019t delete any files unless you run it with the <code>--delete-removed<\/code>\u00a0option.<\/p>\n<p><code>s3cmd sync<\/code>\u00a0won\u2019t run automatically, so if you\u2019d like to keep this directory regularly updated, you\u2019ll need to run this command regularly. You can automate this with <code>cron<\/code>; Open your crontab with <code>crontab -e<\/code>, and add this command to end:<\/p>\n<pre>0 0 * * * s3cmd sync directory s3:\/\/bucket &gt;\/dev\/null 2&gt;&amp;1<\/pre>\n<p>This will sync \u201cdirectory\u201d to \u201cbucket\u201d once a day. By the way, if <code>crontab -e<\/code>\u00a0got you stuck in <code>vim<\/code>, you can change the default text editor with <code>export VISUAL=nano;<\/code>, or whichever you prefer.<\/p>\n<p><code>s3cmd<\/code>\u00a0has a lot of subcommands; you can copy between buckets with <code>cp<\/code>, move files with <code>mv<\/code>, and even create and remove buckets from the command line with <code>mb<\/code>\u00a0and <code>rb<\/code>, respectively. Use <code>s3cmd -h<\/code>\u00a0for a full list.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Another_Option_AWS_CLI\"><\/span>Another Option: AWS CLI<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Beyond <code>s3cmd<\/code>, there are a few other command line options for syncing files to S3. AWS provides their own tools with the AWS CLI. You\u2019ll need Python 3+, and can install the CLI from <code>pip3<\/code>\u00a0with:<\/p>\n<pre>pip3 install awscli --upgrade --user<\/pre>\n<p>This will install the <code>aws<\/code>\u00a0command, which you can use to interact with AWS services. You\u2019ll need to configure it in the same way as s3cmd, which you can do with:<\/p>\n<pre>aws configure<\/pre>\n<p>You\u2019ll be asked to enter the access key and secret key for your IAM user.<\/p>\n<p>The syntax for AWS CLI is similar to <code>s3cmd<\/code>. To upload a file, use:<\/p>\n<pre>aws s3 cp file s3:\/\/bucket<\/pre>\n<p>To sync a whole folder, use:<\/p>\n<pre>aws s3 sync folder s3:\/\/bucket<\/pre>\n<p>You can copy and even sync between buckets with the same commands. You can use <code>aws help<\/code>\u00a0for a full command list, or read the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.aws.amazon.com\/cli\/latest\/reference\/s3\/index.html?tag=reviewgeek-20\">command reference<\/a> on their website.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Full_Backups_Restic_Duplicity\"><\/span>Full Backups: Restic,\u00a0Duplicity<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you want to do large backups, you may want to use another tool rather than a simple sync utility. When you sync to S3 with <code>s3cmd<\/code>\u00a0or the AWS CLI, any changes you\u2019ve made will overwrite the current files. Because the main worry of cloud file storage isn\u2019t usually <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.backblaze.com\/blog\/cloud-storage-durability\/\">drive failure<\/a>, but accidental deletion without access to revision history, this is a problem.<\/p>\n<p>AWS supports <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/dev\/Versioning.html?tag=reviewgeek-20\">file versioning<\/a>, which solves this issue somewhat, but you may still want to use a more powerful backup program to handle it yourself, especially if you\u2019re doing full-drive backups.<\/p>\n<p><a rel=\"nofollow noopener\" target=\"_blank\" href=\"http:\/\/duplicity.nongnu.org\/\">Duplicity<\/a> is a simple utility that backs up files in the form of encrypted TAR volumes. The first archive is a complete backup and then any subsequent archives are incremental, storing only the changes made since the last archive.<\/p>\n<p>This is very efficient, but restoring from a backup is less efficient, as the restoration process will have to follow the chain of changes to arrive at the final state of the data. <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/medium.com\/@denniswebb\/fast-and-secure-backups-to-s3-with-restic-49fd07944304\">Restic<\/a> solves this issue by storing data in deduplicated encrypted blocks, and keeps a snapshot of each version for restoration. This way, the current state of the files is easily referenceable, and each revision is still accessible.<\/p>\n<p>Both tools can be configured to work with AWS S3, as well as multiple other storage providers. Alternatively, if you just want to back up EBS-based EC2 instances, you can use <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/EBSSnapshots.html#how_snapshots_work?tag=reviewgeek-20\">incremental EBS snapshots<\/a>, though it is pricier than backing up manually to S3.\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/577\/how-to-sync-files-from-linux-to-amazon-s3\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Sync Files from Linux to Amazon S3 \u2013 CloudSavvy IT&#8221; AWS S3 is Amazon\u2019s cloud storage service, allowing you to store individual files as objects in a bucket. You can upload files from the command line on your Linux server, or even sync entire directories to S3. If you just want to share&#8230;<\/p>\n","protected":false},"author":1,"featured_media":293557,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2019\/06\/55634f08.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-293556","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/293556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=293556"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/293556\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/293557"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=293556"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=293556"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=293556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}