{"id":236566,"date":"2021-04-27T16:00:16","date_gmt":"2021-04-27T13:00:16","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it\/"},"modified":"2021-04-27T16:00:16","modified_gmt":"2021-04-27T13:00:16","slug":"codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it\/","title":{"rendered":"#Codecov Hacked! What To Do Now if You Use Codecov \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a30bba3d2534\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a30bba3d2534\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it\/#What_is_Codecov\" >What is Codecov?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it\/#Supply-Chain_Attacks\" >Supply-Chain Attacks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it\/#The_Codecov_Attack\" >The Codecov Attack<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/codecov-hacked-what-to-do-now-if-you-use-codecov-cloudsavvy-it\/#Affected_What_You_Should_Do_Now\" >Affected? What You Should Do Now<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#Codecov Hacked! What To Do Now if You Use Codecov \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10853\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/04\/10fb15c7-1.png?width=1200&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Codecov Bash uploader script with line of injected code highlighted\" width=\"1200\" height=\"488\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Codecov was hacked in a way that impacts all of its customers. Thousands of commercial enterprises and open-source projects are affected. Here\u2019s what you need to do if you\u2019re one of them.<\/p>\n<h2 id=\"what-is-codecov\"><span class=\"ez-toc-section\" id=\"What_is_Codecov\"><\/span>What is Codecov?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you\u2019re not involved with\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/en.wikipedia.org\/wiki\/CI\/CD\">continuous integration and continuous deployment\/delivery<\/a>\u00a0(CI\/CD) or other software development automation, you might not be familiar with the name Codecov.<\/p>\n<p>Codecov provides a hosted service that tells developers how much of their source code has been checked by the automated tests that are part of their software build process. It\u2019s a metric called\u00a0<em>code coverage<\/em>. Complicated software projects can have thousands of source code files. Knowing the code coverage tells you how effective your testing actually is because untested code might harbor bugs.<\/p>\n<p>There are many possible paths of execution inside non-trivial code. Writing tests that check all of the paths is difficult. Basically, you\u2019re writing code to find flaws in other code. But if your test code isn\u2019t designed and written correctly it won\u2019t cover all possible execution paths. That means there\u2019ll be holes in your code coverage.<\/p>\n<p>Codecov reads the output generated by your build process and produces reports that show exactly which execution paths your test code has missed. You can add tests to cover those regions or modify the logic of existing tests to make them thoroughly exercise the routine they\u2019re supposed to test.<\/p>\n<p>Code coverage is vital to producing stable software, especially with large teams of developers. You can tell how seriously this is taken by looking at some of Codecov\u2019s 29,000 customers. Organizations like\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/kubernetes.io\/\">Kubernetes<\/a>,\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/mozilla.org\/\">Mozilla<\/a>, <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.hashicorp.com\/\">HashiCorp<\/a>, and\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/atlassian.com\/\">Atlassian<\/a>\u00a0all rely on Codecov for their code coverage reports. And because Codecov is free to open-source projects, there are thousands of open-source projects using Codecov too.<\/p>\n<p>That\u2019s why it is such a good target for cybercriminals.<\/p>\n<h2 id=\"supply-chain-attacks\"><span class=\"ez-toc-section\" id=\"Supply-Chain_Attacks\"><\/span>Supply-Chain Attacks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>A supply-chain attack compromises a supplier or service provider in order to compromise the real targets\u2014all of their customers and users. If you want to poison an entire town you\u2019re not going to go house to house. You\u2019ll poison the water treatment plant and wait.<\/p>\n<p>This is what h<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ened in the recent\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.solarwinds.com\/\">SolarWinds<\/a>\u00a0attack. SolarWinds wasn\u2019t the target, they were just an efficient route for the threat actors to get at all of SolarWinds\u2019 customers. And because Solarwinds\u2019 customers for the most part were managed service providers and outsourced IT support organizations, all of\u00a0<em>their<\/em>\u00a0customers were exposed to risk as well.<\/p>\n<p><strong>RELATED:<\/strong> <strong><em>SolarWinds Hack: What Happened and How To Protect Yourself<\/em><\/strong><\/p>\n<h2 id=\"the-codecov-attack\"><span class=\"ez-toc-section\" id=\"The_Codecov_Attack\"><\/span>The Codecov Attack<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>On April 15, 2021, Codecov\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/about.codecov.io\/security-update\/\">publicly disclosed<\/a>\u00a0that\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.codecov.io\/docs\/about-the-codecov-bash-uploader\">a Bash script<\/a>\u00a0used to upload files to its servers had been modified by threat actors. The initial compromise at Codecov is thought to have occurred in late January 2021. The compromised script collected sensitive information from customers\u2019 continuous integration environments and uploaded that information to the attacker\u2019s server. Access credentials such as ID tokens and API keys as well as anything stored in environment variables were harvested by the modified script.<\/p>\n<p>It was a classic supply-chain attack. By injecting a single line of code into Codecov\u2019s Bash uploader script, the threat actors had the means to access the continuous integration environments of all customers that used the script. And because that script is used in three of Codecov\u2019s uploading routines\u2014Codecov-actions, CircleCl Orb, and Bitrise Step\u2014very few Codecov customers would not be exposed to this risk.<\/p>\n<p>That doesn\u2019t mean all customers have been breached, just that they have been exposed to the threat of unauthorized access. HashiCorp is one customer known to\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/discuss.hashicorp.com\/t\/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure\/23512\">have been compromised<\/a>, while others such as Atlassian and Hewlett Packard Enterprises\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.reuters.com\/technology\/codecov-hackers-breached-hundreds-restricted-customer-sites-sources-2021-04-19\/\">have not discovered any evidence of compromise<\/a>.<\/p>\n<p>The threat actors gained access to the Codecov infrastructure through a poorly configured\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.docker.com\/\">Docker<\/a>\u00a0container. They added a line to the Bash uploader script at line 525 in the file. We can isolate the new line using the\u00a0 <code>colordiff<\/code> <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/linux.die.net\/man\/1\/colordiff\">command<\/a>. The <code>diff<\/code> <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/linux.die.net\/man\/1\/diff\">command<\/a> would work just as well, but the <code>colordiff<\/code> output is a little easier to read, being color-coded. It\u2019s available in all Linux distribution repositories.<\/p>\n<pre>diff current.sh compromised.sh<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10858\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/04\/4a47a0db-2.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"644\" height=\"210\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>The IP address of the attacker\u2019s server was 104.248.94.23, which is a virtual server hosted by\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.anrdoezrs.net\/links\/3607085\/type\/am\/sid\/10851\/https:\/\/www.digitalocean.com\/\">Digital Ocean<\/a>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-10859\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/04\/fb5c81ed-2.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"IP identity for the attacker's server\" width=\"305\" height=\"393\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Needless to say, the cybercriminals won\u2019t use an IP address that points back to themselves.<\/p>\n<p>As soon as Codecov discovered the compromise they took steps to close down the unauthorized access, communicate to their customers, and investigate the incident. Codecov has:<\/p>\n<ul>\n<li>Collaborated with Digital Ocean to have the threat actor\u2019s server taken down.<\/li>\n<li>Regenerated and updated all affected or potentially affected Codecov credentials and closed off the unauthorized access that allowed the uploader script to be modified.<\/li>\n<li>Checked their infrastructure logs to determine how the unauthorized access was possible, and which authentication key was used.<\/li>\n<li>Improved or implemented network monitoring and auditing tools at key points in their infrastructure to detect and prevent a recurrence of this type of attack.<\/li>\n<\/ul>\n<h2 id=\"what-you-should-do\"><span class=\"ez-toc-section\" id=\"Affected_What_You_Should_Do_Now\"><\/span>Affected? What You Should Do Now<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Codecov has emailed all customers they believe are at risk. Whether you have heard from them or not, if your organization is a customer of Codecov you have to assume that you are at risk of compromise.<\/p>\n<p>Because credentials, connection keys, and other sensitive and confidential information are passed from step to step of the CI\/CD process, the threat actors may have harvested those details. They may have been able to gather and retrieve:<\/p>\n<ul>\n<li>Any authentication or privilege credentials, tokens, or keys that were accessible to the script while customer CI\/CD processes were running.<\/li>\n<li>Any third-party services, data, or source code that could be accessed by the CI\/Cd processes.<\/li>\n<li>The git remote repository details.<\/li>\n<\/ul>\n<p>You can easily check what is in the environment variables of your CI\/CD environment by using the\u00a0 <code>env<\/code> <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/linux.die.net\/man\/1\/env\">command<\/a> and piping it into <code>less<\/code> to <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/linux.die.net\/man\/1\/less\">make it more manageable<\/a>:<\/p>\n<pre>env | less<\/pre>\n<ul>\n<li>Review the output and if there is anything that is sensitive or permits any kind of access, change the credentials on that account, platform, or service.<\/li>\n<li>If you are using a local copy of the Bash uploader script you are unlikely to have been affected, but you\u2019re still encouraged to replace that local script with\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/codecov.io\/bash\">the latest version<\/a>.<\/li>\n<li>Audit your system for attempted use by the invalidated credentials and keys. If attempts are detected, it means the threat actors are trying to use the information they\u2019ve exfiltrated from your CI\/CD platform to get in.<\/li>\n<\/ul>\n<p>Act defensively. If there is any suspicion or possibility of compromise, regenerate the relevant credentials and keys im<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a>tely.\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/10851\/codecov-hacked-what-to-do-now-if-you-use-codecov\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#Codecov Hacked! What To Do Now if You Use Codecov \u2013 CloudSavvy IT&#8221; Codecov was hacked in a way that impacts all of its customers. Thousands of commercial enterprises and open-source projects are affected. Here\u2019s what you need to do if you\u2019re one of them. What is Codecov? If you\u2019re not involved with\u00a0continuous integration and&#8230;<\/p>\n","protected":false},"author":1,"featured_media":236567,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/04\/10fb15c7-1.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-236566","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/236566","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=236566"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/236566\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/236567"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=236566"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=236566"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=236566"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}