{"id":381850,"date":"2021-12-16T14:00:57","date_gmt":"2021-12-16T11:00:57","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-migrate-an-aws-s3-bucket-to-another-account-or-service-cloudsavvy-it\/"},"modified":"2021-12-16T14:00:57","modified_gmt":"2021-12-16T11:00:57","slug":"how-to-migrate-an-aws-s3-bucket-to-another-account-or-service-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-migrate-an-aws-s3-bucket-to-another-account-or-service-cloudsavvy-it\/","title":{"rendered":"#How to Migrate an AWS S3 Bucket to Another Account or Service \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a40a99b4a4c5\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a40a99b4a4c5\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-migrate-an-aws-s3-bucket-to-another-account-or-service-cloudsavvy-it\/#How_Does_This_Work\" >How Does This Work?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-migrate-an-aws-s3-bucket-to-another-account-or-service-cloudsavvy-it\/#Setting_Up_rclone\" >Setting Up rclone<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-migrate-an-aws-s3-bucket-to-another-account-or-service-cloudsavvy-it\/#Performing_The_Transfer\" >Performing The Transfer<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Migrate an AWS S3 Bucket to Another Account or Service \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage alignnone size-full wp-image-5269\" data-pagespeed-lazy-srcset=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png?width=398&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 400w, https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 1200w\" sizes=\"auto, 400w, 1200w\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"AWS Logo\" width=\"700\" height=\"300\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>AWS\u2019s Simple Storage Service (S3) is great for storing large amounts of objects, but it\u2019s also an API that\u2019s compatible with many other competiting services. If you want to move off AWS, transferring an S3 bucket is easy to do.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"How_Does_This_Work\"><\/span>How Does This Work?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If both services you\u2019re transerring to and from are S3 compatible, you can simply use a utility like <code>rclone<\/code>, configured to access each service, to transfer all the items over. For example, you could transfer from S3 to Digital Ocean\u2019s compatible Spaces service, or transfer from an S3 bucket in one account to another account.<\/p>\n<p>To handle the transfer, <code>rclone<\/code>\u00a0will read from the source bucket, find all the files that need to be transferred, and handle cloning them into the destination bucket. <code>rclone<\/code>\u00a0can also handle file updates, which can be useful if the source bucket is being written to as it\u2019s transferring.<\/p>\n<p>As far as transfer times go, it will likely take a while depending on the size of the bucket. Number of files is also an issue, as <code>rclone<\/code>\u00a0adds overhead for each transfer. If you have millions of files, or multiple terabytes, you should be prepared for hours of transfer times.<\/p>\n<p>Luckily, you can perform the large initial transfer while the bucket is still actively being written to. You will likely need a bit of downtime to ensure the buckets are synced up before finally switching over. If that is a problem, there are other tools available to seamlessly transfer, including commercial tools like <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/cloud.netapp.com\/cloud-sync-service\">NetApp Cloud Sync<\/a> that can\u00a0sync multiple buckets together.<\/p>\n<p>Some cloud services, like Google Cloud Platform, have services of their own that can handle the transfer. If you\u2019re moving to a platform that supports this, you\u2019ll likely want to use their service instead.<\/p>\n<p><strong>RELATED:<\/strong> <strong><em>How to Transfer an S3 Bucket to Google Cloud Platform Storage<\/em><\/strong><\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Setting_Up_rclone\"><\/span>Setting Up rclone<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The simplest method is to set up <code>rclone<\/code>\u00a0on your own server to handle the transfer operation. You will need to run it in the background, or through a <code>tmux<\/code>\u00a0window so you can disconnect from long transfers.<\/p>\n<p><strong>RELATED:<\/strong> <strong><em>How to Use tmux on Linux for Terminal Multitasking<\/em><\/strong><\/p>\n<p><code>rclone<\/code>\u00a0is available from most package managers:<\/p>\n<pre>apt install rclone -y<\/pre>\n<p><code>rclone<\/code>\u00a0is meant mostly to transfer files locally or between SSH compatible servers, so it will need a bit of configuration to handle transfers between S3 services. This file is located at:<\/p>\n<pre>~\/.config\/rclone\/rclone.conf<\/pre>\n<p>Add a new block with the following configuration, which hooks it up to your AWS account (not a specific bucket):<\/p>\n<pre>[s3]&#13;\ntype = s3&#13;\nenv_auth = false&#13;\nacl = private&#13;\naccess_key_id = ACCESS_KEY&#13;\nsecret_access_key = SECRET_KEY&#13;\nregion = REGION&#13;\nlocation_constraint = LOCATION_CONSTRAINT<\/pre>\n<p>You will need to fill in the config with your access key and secret, and enter in your bucket\u2019s region. You can find a <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.aws.amazon.com\/general\/latest\/gr\/rande.html#s3_region?tag=reviewgeek-20\">list of the regions from the AWS docs<\/a>.<\/p>\n<p>You will need to fill in another block for the other service you\u2019re transfering to. If you\u2019re moving between AWS accounts, you\u2019ll need a separate key with access to that account. If you\u2019re moving to a service like DO Spaces, you\u2019ll need to define another block with a new endpoint configured:<\/p>\n<pre>[spaces]&#13;\ntype = s3&#13;\nenv_auth = false&#13;\nacl = private&#13;\naccess_key_id = ACCESS_KEY&#13;\nsecret_access_key = SECRET_KEY&#13;\nendpoint = nyc3.digitaloceanspaces.com<\/pre>\n<p>In any case, you will need to give it a new name on the block title, because these are two separate remotes.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Performing_The_Transfer\"><\/span>Performing The Transfer<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Once configured, you will be able to view all the possible remotes<\/p>\n<pre>rclone listremotes&#13;\n&#13;\ns3:&#13;\nspaces:<\/pre>\n<p><img decoding=\"async\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/05\/d2bcfb63.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Confirm the type of remote by adding the --long flag to the rclone listremotes command.\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>You can view a bucket\u2019s contents by using the endpoint name followed by a colon and the bucket name.<\/p>\n<pre>rclone tree s3:source-bucket<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15023\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/12\/d222b982.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"500\" height=\"265\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Then, you can run the sync, with some extra flags for optimal performance:<\/p>\n<pre>rclone sync source:\/source-bucket &#13;\ndestination:\/destination-bucket &#13;\n-P -v --log-file \/var\/log\/rclone\/rclone-1.log &#13;\n--create-empty-src-dirs --s3-chunk-size 20M &#13;\n--s3-upload-concurrency 64 --checksum<\/pre>\n<p>The <code>-P<\/code>\u00a0flag will allow you to view progress interactively in your terminal, and will give an estimate of how long it is going to take.<\/p>\n<p><code>rclone sync<\/code>\u00a0will simply scan the source bucket and update the target bucket. You can continue modifying the source bucket while the transfer completes. After it\u2019s done, you can run additional transfers and continue to sync the buckets together.\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/15018\/how-to-migrate-an-aws-s3-bucket-to-another-account-or-service\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Migrate an AWS S3 Bucket to Another Account or Service \u2013 CloudSavvy IT&#8221; AWS\u2019s Simple Storage Service (S3) is great for storing large amounts of objects, but it\u2019s also an API that\u2019s compatible with many other competiting services. If you want to move off AWS, transferring an S3 bucket is easy to do&#8230;.<\/p>\n","protected":false},"author":1,"featured_media":381851,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-381850","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/381850","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=381850"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/381850\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/381851"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=381850"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=381850"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=381850"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}