{"id":309117,"date":"2021-07-27T12:00:08","date_gmt":"2021-07-27T09:00:08","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-use-multi-threaded-processing-in-bash-scripts-cloudsavvy-it\/"},"modified":"2021-07-27T12:00:08","modified_gmt":"2021-07-27T09:00:08","slug":"how-to-use-multi-threaded-processing-in-bash-scripts-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-use-multi-threaded-processing-in-bash-scripts-cloudsavvy-it\/","title":{"rendered":"#How to Use Multi-Threaded Processing in Bash Scripts \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2f1d3563bc5\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2f1d3563bc5\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-multi-threaded-processing-in-bash-scripts-cloudsavvy-it\/#What_Is_multi-threaded_programming\" >What Is multi-threaded programming?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-multi-threaded-processing-in-bash-scripts-cloudsavvy-it\/#Multi-threaded_and_Background_Process_Management\" >Multi-threaded (and Background) Process Management<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-multi-threaded-processing-in-bash-scripts-cloudsavvy-it\/#Wrapping_up\" >Wrapping up<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Use Multi-Threaded Processing in Bash <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">Script<\/a>s \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<figure style=\"width: 1200px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage wp-image-12956 size-full\" srcset=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/07\/ca32dc30.jpg?width=398&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 400w, https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/07\/ca32dc30.jpg?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 1200w\" sizes=\"auto, 400w, 1200w\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/07\/ca32dc30.jpg?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"cat playing with yarn\" width=\"1200\" height=\"675\" data-crediturl=\"https:\/\/www.shutterstock.com\/image-photo\/grey-beautiful-little-kitten-play-many-185435531\" data-credittext=\"Aleksey Mnogosmyslov\/Shutterstock.com\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><figcaption class=\"wp-caption-text\"><span class=\"type:primaryImage imagecredit\"><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.shutterstock.com\/image-photo\/grey-beautiful-little-kitten-play-many-185435531\">Aleksey Mnogosmyslov\/Shutterstock.com<\/a><\/span><\/figcaption><\/figure>\n<p>Multi-threaded programming has always been of interest to developers to increase application performance and optimize resource usage. This guide will introduce you to Bash multi-threaded coding basics.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"What_Is_multi-threaded_programming\"><\/span>What Is <i>multi-threaded programming<\/i>?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>A picture is worth a thousand words, and this holds when it comes to showing the difference between single (1) thread programming and multi-threaded (&gt;1) programming in Bash:<\/p>\n<pre>sleep 1&#13;\nsleep 1 &amp; sleep 1&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-12278 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/06\/bf3f4bca.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"(Last command) A simple two thread one-line command which will run two sleep processes in parallel, one in the background \" width=\"342\" height=\"84\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Our first multi-threaded programming setup or mini one-liner script could not have been simpler; in the first line, we sleep for one second using the <code>sleep 1<\/code> command. As far as the user is concerned, a single thread was executing a single sleep of one second.<\/p>\n<p>In the second line, we have two one-second sleep commands. We join them by using a <code>&amp;<\/code> separator, which does not only act as a separator between the two <code>sleep<\/code> commands, but also as an indicator to Bash to start the first command in a background thread.<\/p>\n<p>Normally, one would terminate a command by using a semicolon (<code>;<\/code>). Doing so would execute the command and only then proceed to the next command listed behind the semicolon. For example, executing <code>sleep 1; sleep 1<\/code> would take just over two seconds \u2013 exactly one second for the first command, one second for the second, and a tiny amount of system overhead for each of the two commands.<\/p>\n<p>However, instead of terminating a command with a semicolon one can use other command terminators which Bash recognizes like <code>&amp;<\/code>, <code>&amp;&amp;<\/code> and <code>||<\/code>. The <code>&amp;&amp;<\/code> syntax is quite unrelated to multi-threaded programming, it simply does this; proceed with executing the second command only if the first command was successful. The <code>||<\/code> is the opposite of <code>&amp;&amp;<\/code> and will execute the second command only if the first command failed.<\/p>\n<p>Returning to multi-threaded programming, using <code>&amp;<\/code> as our command terminator will initiate a background process executing the command preceding it. It then im<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a>tely proceeds with executing the next command in the current shell while leaving the background process (thread) to execute by itself.<\/p>\n<p>In the output of the command we can see a background process being started (as indicated by <code>[1] 445317<\/code> where <code>445317<\/code> is the Process ID or PID of the just started background process and <code>[1]<\/code> is an indicated that this is our first background process) and it subsequently being terminated (as indicated by <code>[1]+ Done sleep 1<\/code>).<\/p>\n<p>If you would like to view an additional example of background process handling, please see our Bash Automation and Scripting Basics (Part 3) article. Additionally, Bash Process Termination Hacks may be of interest.<\/p>\n<p>Let\u2019s now proof that we are effectively running two <code>sleep<\/code> processes at the same time:<\/p>\n<pre>time sleep 1; echo 'done'&#13;\ntime $(sleep 1 &amp; sleep 1); echo 'done'&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-12279 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/06\/878db57f.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Running two sleep threads in parallel, with one in the background, using a subshell\" width=\"541\" height=\"222\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Here we start our <code>sleep<\/code> process under <code>time<\/code> and we can see how our single threaded command ran for exactly 1.003 seconds before our command line prompt was returned.<\/p>\n<p>However, in the second example, it took about the same time (1.005 seconds) even though we were executing two periods (and processes) of sleep, though not consecutively. Again we used a background process for the first sleep command, leading to (semi-)parallel execution, i.e., multi-threaded.<\/p>\n<p>We also used a subshell wrapper (<code>$(...)<\/code>) around our two sleep commands to combine them together under <code>time<\/code>. As we can see our <code>done<\/code> output shows in 1.005 seconds and thus the two <code>sleep 1<\/code> commands must have run simultaneously. Interesting is the very small increase in overall processing time (0.002 seconds) which can be easily explained by the time required to start a subshell and the time required to initiate a background process.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Multi-threaded_and_Background_Process_Management\"><\/span>Multi-threaded (and Background) Process Management<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In Bash, multi-threaded coding will normally involve background threads from a main one-line script or full Bash script. In essence, one may think about multi-threaded coding in Bash as starting several background threads. When one starts to code using multiple threads, it quickly becomes clear that such threads will usually require some handling. For example, take the fictive example where we start five concurrent periods (and processes) of sleep in a Bash script;<\/p>\n<pre>#!\/bin\/bash&#13;\n&#13;\nsleep 10 &amp; &#13;\nsleep 600 &amp; &#13;\nsleep 1200 &amp; &#13;\nsleep 1800 &amp; &#13;\nsleep 3600 &amp;&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-12280 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/06\/8706fd84.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Running five parallel sleep threads in the background from a script\" width=\"234\" height=\"189\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>When we start the script (after making it executable using <code>chmod +x rest.sh<\/code>), we see no output! Even if we execute <code>jobs<\/code> (the command which shows any background jobs in progress), there is no output. Why?<\/p>\n<p>The reason is that the shell which was used to start this script (i.e., the current shell) is not the same shell (nor the same thread; to start thinking in terms of subshells as threads in and by themselves) that executed the actual sleep commands or placed them into the background. It was rather the (sub)shell which was started when <code>.\/rest.sh<\/code> was executed.<\/p>\n<p>Let\u2019s change our script by adding <code>jobs<\/code> inside the script. This will ensure that <code>jobs<\/code> is executed from within the (sub)shell where it is relevant, the same one as to where the periods (and processes) of sleep were started.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-12281\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/06\/68d0e004.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Adding jobs command in the script\" width=\"393\" height=\"287\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>This time we can see the list of background processes being started thanks to the <code>jobs<\/code> command at the end of the script. We can also see their PID\u2019s (Process Identifiers). These PIDs are very important when it comes to handling and managing background processes.<\/p>\n<p>Another way to obtain the background Process Identifier is to query for it immediately after placing a program\/process into the background:<\/p>\n<pre>#!\/bin\/bash&#13;\n&#13;\nsleep 10 &amp; &#13;\necho ${!}&#13;\nsleep 600 &amp; &#13;\necho ${!}&#13;\nsleep 1200 &amp; &#13;\necho ${!}&#13;\nsleep 1800 &amp; &#13;\necho ${!}&#13;\nsleep 3600 &amp;&#13;\necho ${!}&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-12282\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/06\/4d77f777.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Quering the PID (Process ID) of the last background process to be started using ${!}\" width=\"325\" height=\"343\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Similar to our <code>jobs<\/code> command (with new PID\u2019s now as we restarted our <code>rest.sh<\/code> script), thanks to the Bash <code>${!}<\/code> variable being echoed, we will now see the five PID\u2019s being displayed almost immediately after the script starts: the various sleep processes were placed into background threads one after the other.<\/p>\n<p><code>The <i>wait<\/i> Command<\/code><\/p>\n<p>Once we have started our background processes, we have nothing further to do than wait for them to be finished. However, when each background process is executing a complex subtask, and we need the main script (which started the background processes) to resume executing when one or more of the background processes terminates, we need additional code to handle this.<\/p>\n<p>Let\u2019s expand our script now with the <code>wait<\/code> command to handle our background threads:<\/p>\n<pre>#!\/bin\/bash&#13;\n&#13;\nsleep 10 &amp; &#13;\nT1=${!}&#13;\nsleep 600 &amp; &#13;\nT2=${!}&#13;\nsleep 1200 &amp; &#13;\nT3=${!}&#13;\nsleep 1800 &amp; &#13;\nT4=${!}&#13;\nsleep 3600 &amp;&#13;\nT5=${!}&#13;\n&#13;\necho \"This script started 5 background threads which are currently executing with PID's ${T1}, ${T2}, ${T3}, ${T4}, ${T5}.\"&#13;\nwait ${T1}&#13;\necho \"Thread 1 (sleep 10) with PID ${T1} has finished!\"&#13;\nwait ${T2}&#13;\necho \"Thread 2 (sleep 600) with PID ${T2} has finished!\"&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-12283\" src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/06\/c2b5fbd9.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"1000\" height=\"393\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Here we expanded our script with two <code>wait<\/code> commands which wait for the PID attached to the first and second threads to terminate. After 10 seconds, our first thread exists, and we are notified of the same. Step by step, this script will do the following: start five threads at almost the same time (though the starting of the threads itself is still sequential and not parallel) where each of the five <code>sleep<\/code>\u2018s will execute in parallel.<\/p>\n<p>The main script then (sequentially) reports on the thread created and subsequently waits for the Process ID of the first thread to terminate. When that happens, it will sequentially report on the first thread finishing and commence a wait for the second thread to finish, etc.<\/p>\n<p>Using the Bash idioms <code>&amp;<\/code>, <code>${!}<\/code> and the <code>wait<\/code> command give us great flexibility when it comes to running multiple threads in parallel (as background threads) in Bash.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Wrapping_up\"><\/span>Wrapping up<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In this article we explored Bash multi-threaded scripting basics. We introduced the background process operator (<code>&amp;<\/code>) using some easy-to-follow examples showing both single and multi-threaded <code>sleep<\/code> commands. Next, we looked at how to handle background processes via the commonly used Bash idioms <code>${!}<\/code> and <code>wait<\/code>. We also explored the <code>jobs<\/code> command to see running background threads\/processes.<\/p>\n<p>If you enjoyed reading this article, have a look at our Bash Process Termination Hacks article. <strong>Enjoy!<\/strong>\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/12277\/how-to-use-multi-threaded-processing-in-bash-scripts\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Use Multi-Threaded Processing in Bash Scripts \u2013 CloudSavvy IT&#8221; Aleksey Mnogosmyslov\/Shutterstock.com Multi-threaded programming has always been of interest to developers to increase application performance and optimize resource usage. This guide will introduce you to Bash multi-threaded coding basics. What Is multi-threaded programming? A picture is worth a thousand words, and this holds when&#8230;<\/p>\n","protected":false},"author":1,"featured_media":309118,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/07\/ca32dc30.jpg","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-309117","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/309117","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=309117"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/309117\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/309118"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=309117"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=309117"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=309117"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}