{"id":365841,"date":"2021-11-12T04:59:00","date_gmt":"2021-11-12T01:59:00","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/"},"modified":"2021-11-12T04:59:00","modified_gmt":"2021-11-12T01:59:00","slug":"how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/","title":{"rendered":"#How to Copy Data between AWS and Azure Buckets Using rclone \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a29e8ce8c621\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a29e8ce8c621\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/#Prerequisites\" >Prerequisites<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/#How_to_Set_Up_rclone\" >How to Set Up rclone<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/#Azure\" >Azure<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/#AWS_S3\" >AWS S3<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/#Running_rclone\" >Running rclone<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone-cloudsavvy-it\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Copy Data between AWS and Azure Buckets Using rclone \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage alignnone size-full wp-image-4724\" srcset=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/04\/30fdf294.png?width=398&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 400w, https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/04\/30fdf294.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 1200w\" sizes=\"auto, 400w, 1200w\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/04\/30fdf294.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"azure logo\" width=\"1400\" height=\"600\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Data is one of the most <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/redirect.viglink.com\/?key=204a528a336ede4177fff0d84a044482&amp;u=https%3A%2F%2Fwww.economist.com%2Fleaders%2F2017%2F05%2F06%2Fthe-worlds-most-valuable-resource-is-no-longer-oil-but-data\">valuable commodities<\/a> in the world, and it\u2019s not hard to see why. From marketing to genomics, the analysis of large sets of data leads to predictive models, which steer to favorable outcomes for the business. The more data you use, the better those models are, which means the better outcomes they can produce. Of course, this means that moving data from one place to another is a crucial skill to have for any engineer, but it\u2019s not always as easy as it sounds.<\/p>\n<p>For example, if you use AWS S3 bucket storage, then moving data to another S3 bucket is a single CLI command, <code>aws s3 cp s3:\/\/SourceBucket\/* s3:\/\/DestinationBucket\/<\/code>. Moving those same files to a different cloud provider, like Microsoft Azure or Google Cloud Platform, requires an entirely different tool.<\/p>\n<p>By the end of this tutorial, you\u2019ll be able to sync files from an AWS S3 bucket to an Azure blob storage container using <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/rclone.org\/\">rclone<\/a>, an open-source data synchronization tool that works with most cloud providers and local file systems.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Prerequisites\"><\/span>Prerequisites<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To follow along, you\u2019ll need the following:<\/p>\n<ul>\n<li>An AWS S3 bucket<\/li>\n<li>An Azure blob storage container<\/li>\n<li>AWS access keys and Azure storage account access keys<\/li>\n<li>A computer running any modern operating system\n<ul>\n<li>Screenshots are from Windows 10 with WSL<\/li>\n<\/ul>\n<\/li>\n<li>Some files to copy<\/li>\n<\/ul>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"How_to_Set_Up_rclone\"><\/span>How to Set Up rclone<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/rclone.org\/install\/\">Installing rclone<\/a> is different for each operating system, but once it\u2019s installed, the instructions are the same: run <code>rclone config<\/code><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"imgchk9 alignnone wp-image-4851 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/05\/aadcc5eb-1.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"A successful rclone installation.\" width=\"808\" height=\"168\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Running the config command will prompt you to link the accounts of your cloud providers to rclone. The rclone term for this is a <em>remote. <\/em>When you run the config command, enter <code>n<\/code> to create a new remote. You\u2019ll need one for both AWS and Azure, but there are several other providers to choose from as well.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"imgchk9 alignnone wp-image-4852 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/05\/774fe9f6.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Linking the accounts of your cloud providers to rclone. \" width=\"772\" height=\"681\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Azure\"><\/span>Azure<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>After choosing Azure blob storage, you\u2019ll need:<\/p>\n<ul>\n<li>A name for the remote. (In this demo, it\u2019s \u201cAzure.\u201d)<\/li>\n<li>The storage account\u2019s name<\/li>\n<li>One of the storage account access keys<\/li>\n<\/ul>\n<p>You\u2019ll be prompted for a Shared Access Signature URL, and while it\u2019s possible to set up using that, this demo is just using an access key. After entering default for the rest of the values by hitting <em>Enter<\/em> through the rest of the setup, you should be able to start using your remote.<\/p>\n<p>To list the remotes configured on your system, enter <code>rclone listremotes<\/code>, which will show the remotes available. You can also list any blob storage containers by running <code>rclone lsd &lt;remote_name&gt;:<\/code>. Make sure to include a <code>:<\/code> at the end of the remote when running these commands because that is how rclone determines if you want to use a remote or not. You can run <code>rclone --help<\/code> at anytime to get the list of available commands.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"imgchk9 alignnone wp-image-4853 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/05\/01b3601c.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"526\" height=\"248\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Using the <code>ls<\/code> and <code>cat<\/code> commands with an rclone remote.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"AWS_S3\"><\/span>AWS S3<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Setting up the remote for an S3 bucket is very similar to the Azure blob storage container, just with a few small differences. Because there are other cloud storage providers that are considered S3 compatible by rclone, you may also get a few extra prompts when running <code>rclone config<\/code>. You\u2019ll need:<\/p>\n<ul>\n<li>A name for the remote. (In this demo, it\u2019s \u201cAWS.\u201d)<\/li>\n<li>An AWS access key and corresponding secret access key<\/li>\n<li>The AWS region that the bucket is in<\/li>\n<\/ul>\n<p>The rest of the prompts can be configured to create other buckets or perform other operations, but for copy, you can skip the rest of them by hitting <em>Enter.<\/em><\/p>\n<p>If the user the access keys belong to has access to the bucket, you will have access to it with the same commands you used to access the Azure remote.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"imgchk9 alignnone wp-image-4854 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/05\/d2bcfb63.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Confirm the type of remote by adding the --long flag to the rclone listremotes command.\" width=\"547\" height=\"263\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>You can confirm the type of remote by adding the <code>--long<\/code> flag to the <code>rclone listremotes<\/code> command.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Running_rclone\"><\/span>Running rclone<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Now that the remotes have been configured, you can transfer files, create new buckets, or manipulate the files in any way you need to using a standard set of commands. Instead of relying on knowing how to work with the AWS S3 CLI or Azure PowerShell, you can communicate between both storage buckets with rclone.<\/p>\n<p>Some common useful commands to get you started are:<\/p>\n<ul>\n<li><code>rclone tree &lt;Remote&gt;:&lt;BucketName&gt;<\/code>\u2014Lists the content of the bucket in a tree format. Add the <code>-C<\/code> flag to add color to the output.<\/li>\n<li><code>rclone size &lt;Remote&gt;:&lt;BucketName&gt;<\/code>\u2014Displays the number of files in the bucket and their total size.<\/li>\n<li><code>rclone sync &lt;Source&gt; &lt;Target&gt;<\/code>\u2014Makes the target match the source, but only modifies the target. Source and target can be either local file paths or rclone remote paths. Add the <code>-P<\/code> flag to view progress interactively.<\/li>\n<\/ul>\n<p>In the following example, the AWS S3 bucket is synced to the Azure remote, which deletes the existing file in Azure and then copies the data from S3. If you need to keep the files in the target folder, then use the <code>rclone copyto<\/code> command.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"imgchk9 alignnone wp-image-4855 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/05\/1ea2463d.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"The AWS S3 bucket is synced to the Azure remote, which deletes the existing file in Azure and then copies the data from S3.\" width=\"736\" height=\"624\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>By now, you should be comfortable installing rclone and configuring remotes, as well as using those remotes to copy data between different clouds. rclone is an extremely flexible tool and isn\u2019t just limited to AWS and Azure, so if you use another cloud provider, try setting up remotes for them as well.\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/4849\/how-to-copy-data-between-aws-and-azure-buckets-using-rclone\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Copy Data between AWS and Azure Buckets Using rclone \u2013 CloudSavvy IT&#8221; Data is one of the most valuable commodities in the world, and it\u2019s not hard to see why. From marketing to genomics, the analysis of large sets of data leads to predictive models, which steer to favorable outcomes for the business&#8230;.<\/p>\n","protected":false},"author":1,"featured_media":365842,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/04\/30fdf294.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-365841","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/365841","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=365841"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/365841\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/365842"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=365841"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=365841"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=365841"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}