{"id":263919,"date":"2021-06-01T17:00:19","date_gmt":"2021-06-01T14:00:19","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-use-aws-object-lambda-to-transform-s3-objects-on-request-cloudsavvy-it\/"},"modified":"2021-06-01T17:00:19","modified_gmt":"2021-06-01T14:00:19","slug":"how-to-use-aws-object-lambda-to-transform-s3-objects-on-request-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-use-aws-object-lambda-to-transform-s3-objects-on-request-cloudsavvy-it\/","title":{"rendered":"#How to Use AWS Object Lambda to Transform S3 Objects on Request \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a33a159c7514\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a33a159c7514\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-aws-object-lambda-to-transform-s3-objects-on-request-cloudsavvy-it\/#What_Is_Object_Lambda\" >What Is Object Lambda?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-aws-object-lambda-to-transform-s3-objects-on-request-cloudsavvy-it\/#Using_Object_Lambda\" >Using Object Lambda<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Use AWS Object Lambda to Transform S3 Objects on Request \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage alignnone size-full wp-image-5269\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"AWS Logo\" width=\"700\" height=\"300\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Object Lambda lets you put a Lambda function in front of S3 objects, allowing them to be transformed on request by your own custom code. Since it runs automatically on Lambda, you don\u2019t have to worry about running your own proxy layer.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"What_Is_Object_Lambda\"><\/span>What Is Object Lambda?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Object Lambda basically takes the place of an API in front of S3. Previously, you\u2019d have to set up a proxy layer on your own infrastructure to handle transforming objects on request. This adds complexity, so AWS added a better solution.<\/p>\n<p><strong>RELATED:<\/strong> <strong><em>What Are Lambda Functions, And How Do You Use Them?<\/em><\/strong><\/p>\n<p>Instead of accessing objects directly, you\u2019ll do so through an Object Lambda Access Point. When you make a GET request for a file in an S3 bucket, the Lambda function for that access point will be automatically called, allowed to access the original object, and return a transformed object back to the <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>lication.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-11547 size-full\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/05\/1f26c206.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"When you make a GET request for a file in an S3 bucket, the Lambda function for that access point will be automatically called, allowed to access the original object, and return a transformed object back to the application.\" width=\"700\" height=\"320\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>The uses for this can be basic, like redacting info or converting JSON to XML, but since it\u2019s your own code, you can do whatever you\u2019d like. You could, for example, run a database lookup and return a transformed object with new data, or make requests to external APIs.<\/p>\n<p>You can have multiple access points per bucket, which can each represent multiple \u201cviews\u201d of the underlying data. To use different access points, you won\u2019t need to update any client code. Simply change the bucket name to the ARN of the Object Lambda Access Point.<\/p>\n<pre>s3<span class=\"token punctuation\">.<\/span>get_object<span class=\"token punctuation\">(<\/span> &#13;\n    Bucket<span class=\"token operator\">=<\/span><span class=\"token string\">'arn:aws:s3-object-lambda:us-east-1:123412341234:accesspoint\/myolap'<\/span><span class=\"token punctuation\">,<\/span> &#13;\n    Key<span class=\"token operator\">=<\/span><span class=\"token string\">'s3.txt' <\/span><span class=\"token punctuation\">)<\/span><\/pre>\n<p>You also don\u2019t need to access the original object by the exact name. For example, your application could request <code>picture_1920x1080.jpg<\/code>, which would find <code>picture.jpg<\/code>\u00a0and resize it to the given dimensions. In this case, the Lambda function would need extra permissions to access the bucket contents.<\/p>\n<p>Of course, you\u2019ll need to pay for all the time spent running Lambda functions. If you\u2019re running a lot of functions through a user-facing access point, this could start to add up. If your transformations are static, you might want to consider caching the objects in a separate S3 bucket. For example, if you have a function that applies filters\/compression to an image, you might want to cache the results instead of rebuilding on every request. For things that depend on external state, though, this won\u2019t be possible.<\/p>\n<p><strong>RELATED:<\/strong> <strong><em>How To Backup an S3 Bucket (And Why You&#8217;d Even Want To)<\/em><\/strong><\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Using_Object_Lambda\"><\/span>Using Object Lambda<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Head over to the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/s3.console.aws.amazon.com\/s3\/?tag=reviewgeek-20\">S3 Management Console<\/a> to get started. Each Object Lambda Access Point needs a regular access point behind it. You\u2019ll need to create this from Access Points &gt; Create in the sidebar.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-11550 size-full\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/05\/e30b36da.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Each Object Lambda Access Point needs a regular access point behind it. You'll need to create this from Access Points &gt; Create in the sidebar.\" width=\"700\" height=\"316\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Enter a name and select a bucket, and make sure to select \u201cInternet\u201d unless this bucket is limited to a single VPC. Once it\u2019s created, copy the ARN for the access point.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-11551 size-full\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/05\/67917dfa.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Enter a name and select a bucket, and make sure to select &quot;Internet&quot; unless this bucket is limited to a single VPC. Once it's created, copy the ARN for the access point.\" width=\"700\" height=\"319\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Create an Object Lambda Access Point:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-11552 size-full\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/05\/6f742f61.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Create an Object Lambda Access Point:\" width=\"700\" height=\"252\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Give it a name and paste it in the ARN of the access point, and the console should display the name of the underlying bucket.<\/p>\n<p>At this point, you\u2019ll need to select a Lambda function. If you have one prepared, you can enter the ARN or select it from the list. Otherwise, you\u2019ll need to head over to the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/console.aws.amazon.com\/lambda\/home?tag=reviewgeek-20\">Lambda Management Console<\/a> to create one.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-11554 size-full\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/05\/37876a9e.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"If you have one prepared, you can enter the ARN or select it from the list. Otherwise, you'll need to head over to the Lambda Management Console to create one.\" width=\"700\" height=\"336\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>At this point, the code is up to you, although AWS provides the following example, which takes the original object and transforms it to uppercase. No matter what language you end up using, you\u2019ll need to grab the event context, make a request to S3 using the URL, transform the object, and then write the response using the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/API\/API_WriteGetObjectResponse.html?tag=reviewgeek-20\">new WriteGetObjectResponse API<\/a>, returning an HTTP status code afterward.<\/p>\n<pre><span class=\"token keyword\">import<\/span> boto3&#13;\n<span class=\"token keyword\">import<\/span> requests&#13;\n&#13;\n<span class=\"token keyword\">def<\/span> <span class=\"token function\">lambda_handler<\/span><span class=\"token punctuation\">(<\/span>event<span class=\"token punctuation\">,<\/span> context<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">:<\/span>&#13;\n    <span class=\"token keyword\">print<\/span><span class=\"token punctuation\">(<\/span>event<span class=\"token punctuation\">)<\/span>&#13;\n&#13;\n    object_get_context <span class=\"token operator\">=<\/span> event<span class=\"token punctuation\">[<\/span><span class=\"token string\">\"getObjectContext\"<\/span><span class=\"token punctuation\">]<\/span>&#13;\n    request_route <span class=\"token operator\">=<\/span> object_get_context<span class=\"token punctuation\">[<\/span><span class=\"token string\">\"outputRoute\"<\/span><span class=\"token punctuation\">]<\/span>&#13;\n    request_token <span class=\"token operator\">=<\/span> object_get_context<span class=\"token punctuation\">[<\/span><span class=\"token string\">\"outputToken\"<\/span><span class=\"token punctuation\">]<\/span>&#13;\n    s3_url <span class=\"token operator\">=<\/span> object_get_context<span class=\"token punctuation\">[<\/span><span class=\"token string\">\"inputS3Url\"<\/span><span class=\"token punctuation\">]<\/span>&#13;\n&#13;\n    &#13;\n    response <span class=\"token operator\">=<\/span> requests<span class=\"token punctuation\">.<\/span>get<span class=\"token punctuation\">(<\/span>s3_url<span class=\"token punctuation\">)<\/span>&#13;\n    original_object <span class=\"token operator\">=<\/span> response<span class=\"token punctuation\">.<\/span>content<span class=\"token punctuation\">.<\/span>decode<span class=\"token punctuation\">(<\/span><span class=\"token string\">'utf-8'<\/span><span class=\"token punctuation\">)<\/span>&#13;\n&#13;\n    &#13;\n    transformed_object <span class=\"token operator\">=<\/span> original_object<span class=\"token punctuation\">.<\/span>upper<span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span>&#13;\n&#13;\n    &#13;\n    s3 <span class=\"token operator\">=<\/span> boto3<span class=\"token punctuation\">.<\/span>client<span class=\"token punctuation\">(<\/span><span class=\"token string\">'s3'<\/span><span class=\"token punctuation\">)<\/span>&#13;\n    s3<span class=\"token punctuation\">.<\/span>write_get_object_response<span class=\"token punctuation\">(<\/span>&#13;\n        Body<span class=\"token operator\">=<\/span>transformed_object<span class=\"token punctuation\">,<\/span>&#13;\n        RequestRoute<span class=\"token operator\">=<\/span>request_route<span class=\"token punctuation\">,<\/span>&#13;\n        RequestToken<span class=\"token operator\">=<\/span>request_token<span class=\"token punctuation\">)<\/span>&#13;\n&#13;\n    <span class=\"token keyword\">return<\/span> <span class=\"token punctuation\">{<\/span><span class=\"token string\">'status_code'<\/span><span class=\"token punctuation\">:<\/span> <span class=\"token number\">200<\/span><span class=\"token punctuation\">}<\/span><\/pre>\n<p>The event object that Lambda receives will look something like this:<\/p>\n<pre>{&#13;\n    \"xAmzRequestId\": \"1a5ed718-5f53-471d-b6fe-5cf62d88d02a\",&#13;\n    \"getObjectContext\": {&#13;\n        \"inputS3Url\": \"https:\/\/myap-123412341234.s3-accesspoint.us-east-1.amazonaws.com\/s3.txt?X-Amz-Security-Token=...\",&#13;\n        \"outputRoute\": \"io-iad-cell001\",&#13;\n        \"outputToken\": \"...\"&#13;\n    },&#13;\n    \"configuration\": {&#13;\n        \"accessPointArn\": \"arn:aws:s3-object-lambda:us-east-1:123412341234:accesspoint\/myolap\",&#13;\n        \"supportingAccessPointArn\": \"arn:aws:s3:us-east-1:123412341234:accesspoint\/myap\",&#13;\n        \"payload\": \"test\"&#13;\n    },&#13;\n    \"userRequest\": {&#13;\n        \"url\": \"\/s3.txt\",&#13;\n        \"headers\": {&#13;\n            \"Host\": \"myolap-123412341234.s3-object-lambda.us-east-1.amazonaws.com\",&#13;\n            \"Accept-Encoding\": \"identity\",&#13;\n            \"X-Amz-Content-SHA256\": \"e3b0c44297fc1c149afbf4c8995fb92427ae41e4649b934ca495991b7852b855\"&#13;\n        }&#13;\n    },&#13;\n    \"userIdentity\": {&#13;\n        \"type\": \"IAMUser\",&#13;\n        \"principalId\": \"...\",&#13;\n        \"arn\": \"arn:aws:iam::123412341234:user\/myuser\",&#13;\n        \"accountId\": \"123412341234\",&#13;\n        \"accessKeyId\": \"...\"&#13;\n    },&#13;\n    \"protocolVersion\": \"1.00\"&#13;\n}<\/pre>\n<p>There are two important pieces of info here\u2014the <code>userRequest<\/code> section, which contains info about the initial request, like URL and HTTP headers, and the <code>userIdentity<\/code>\u00a0section, which can be used to personalize the response based on IAM user.<\/p>\n<p><strong>RELATED:<\/strong> <strong><em>AWS IAM Users Versus. IAM Roles: Which One Should You Use?<\/em><\/strong>\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/11323\/how-to-use-aws-object-lambda-to-transform-s3-objects-on-request\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Use AWS Object Lambda to Transform S3 Objects on Request \u2013 CloudSavvy IT&#8221; Object Lambda lets you put a Lambda function in front of S3 objects, allowing them to be transformed on request by your own custom code. Since it runs automatically on Lambda, you don\u2019t have to worry about running your own&#8230;<\/p>\n","protected":false},"author":1,"featured_media":263920,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/06\/e601b806.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-263919","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/263919","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=263919"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/263919\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/263920"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=263919"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=263919"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=263919"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}