{"id":115583,"date":"2020-11-19T17:00:20","date_gmt":"2020-11-19T14:00:20","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it\/"},"modified":"2020-11-19T17:00:20","modified_gmt":"2020-11-19T14:00:20","slug":"using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it\/","title":{"rendered":"#Using xargs in Combination With bash -c to Create Complex Commands \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a3dfcb1a102f\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a3dfcb1a102f\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it\/#xargs_Introduction\" >xargs Introduction<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it\/#Pipes_and_xargs\" >Pipes and xargs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it\/#Welcome_to_xargs_With_bash_-c\" >Welcome to xargs With bash -c<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/using-xargs-in-combination-with-bash-c-to-create-complex-commands-cloudsavvy-it\/#Wrapping_up\" >Wrapping up<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#Using xargs in Combination With bash -c to Create Complex Commands \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<figure id=\"attachment_8058\" style=\"width: 700px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8058 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/3a3d45977be3d8c4838821ab00408875\/p\/uploads\/2020\/11\/d238ed4a.png\" alt=\"\" width=\"700\" height=\"300\" data-crediturl=\"https:\/\/www.shutterstock.com\/image-photo\/fireworks-2528431\" data-credittext=\"Shutterstock\/ksb\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><figcaption class=\"wp-caption-text\"><span class=\"imagecredit\"><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.shutterstock.com\/image-photo\/fireworks-2528431\">Shutterstock\/ksb<\/a><\/span><\/figcaption><\/figure>\n<p>xargs is fireworks for your shell commands. Any output, generated from any shell command, can be sent to xargs for it to be processed further into another command line. Learn to harness this great power of xargs today!<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"xargs_Introduction\"><\/span>xargs Introduction<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>My colleague Dave McKay wrote an interesting article <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.howtogeek.com\/435164\/how-to-use-the-xargs-command-on-linux\/\">How to Use the xargs Command on Linux<\/a>, which you may like to read first for an detailed introduction and exploration of xargs in <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/general\/\" data-internallinksmanager029f6b8e52c=\"3\" title=\"General\" target=\"_blank\" rel=\"noopener\">general<\/a>.<\/p>\n<p>This article will focus on a specific problem: what to do when you run into limitations of traditional pipe-based stacking, or normal (<code>;<\/code> delimited) stacking of Linux commands, and when even using xargs does not im<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a>tely seem to provide an answer?<\/p>\n<p>This is regularly, if not often, the case when writing one-liner <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">script<\/a>s at the command line, and\/or when processing complex data or data structures. The information presented here is based on research and many years of experience using this particular method and approach.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Pipes_and_xargs\"><\/span>Pipes and xargs<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Anyone learning Bash will grow in their command line scripting skills over time. The great thing with Bash is that any skills learned at the command line easily translate into Bash scripts (which generally are marked with a <code>.sh<\/code> suffix). The syntax is as good as identical.<\/p>\n<p>And, as skills improve, newer engineers will usually discover the Pipe idiom first. Using a pipe is easy and straightforward: the output from the previous command is \u2018piped\u2019 to the input for the next command think about it like a water pipe bringing water from an output source to an input source elsewhere, for example water from a dam piped to a water turbine.<\/p>\n<p>Let\u2019s look at an easy example:<\/p>\n<pre>echo 'a' | sed 's\/a\/b\/'<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-7985\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/f87bd5a61deb62e3487f05292bc9905c\/p\/uploads\/2020\/11\/114f079b-1.png\" alt=\"A simple Bash pipe example with sed regex text replacement\" width=\"252\" height=\"38\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Here we simply echoed \u2018a\u2019 and subsequently changed the same using the text stream editor <i>sed<\/i>. The output is naturally \u2018b\u2019: the sed substituted (s command) \u2018a\u2019 to \u2018b\u2019.<\/p>\n<p>Some time later the engineer will realize that pipes still only have limited capabilities, especially when one wants to pre-process data into a format ready for the next tool. For example, consider this situation:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-7986 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/f52d1b7ad8a36e0bdf4d381402ae103c\/p\/uploads\/2020\/11\/ff2a6343-1.png\" alt=\"Running into challenges using a pipe in combination with kill\" width=\"864\" height=\"109\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Here we start a sleep in the background. Next we use <code>pidof<\/code> to retrieve the PID of the sleep command being executed, and attempt to kill it with <code>kill -9<\/code> (Think of -9 as a destructive mode to kill a process). It fails. We then try to use the PID provided by the shell when we started the background process, but this similarly fails.<\/p>\n<p>The problem is that <code>kill<\/code> does not accept the input directly, whether it comes from <code>ps<\/code> or even from a simple <code>echo<\/code>. To fix this problem, we can use xargs to take the output from either the <code>ps<\/code> or the <code>echo<\/code> command) and provide them as input to <code>kill<\/code>, by making them arguments to the kill command. It is thus as if we executed <code>kill -9 some_pid<\/code> directly. Let\u2019s see how this works:<\/p>\n<pre>sleep 300 &amp;&#13;\npidof sleep | xargs kill -9<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-7987 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/5677638d9282ee1df2ffe5c7c314ffe3\/p\/uploads\/2020\/11\/dac990e0-1.png\" alt=\"xargs resolving the pipe kill command issue seen earlier\" width=\"368\" height=\"39\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>This works perfectly, and achieves what we set out to do: kill the sleep process. One small change to code (i.e. just add xargs in front of the command), yet one big change to how useful Bash can be for the developing engineer!<\/p>\n<p>We can also use the <code>-I<\/code> option (defining the argument replace string) to <code>kill<\/code> to make it a little clearer how we are passing arguments to kill: i12b Here, we define <code>{}<\/code> as our replace string. In other words, whenever xargs will see <code>{}<\/code>, it will substitute <code>{}<\/code> to whatever input it received from the last command.<\/p>\n<p>Still, even this has it\u2019s limitations. How about if we wanted to provide some nice debug information printed inline and from within the statement? It seems impossible thus far.<\/p>\n<p>Yes, we could post-process the output with a <code>sed<\/code> regex, or insert a subshell (<code>$()<\/code>) somewhere, but all these still would seem to have limitations, and especially so when it becomes time to build complex data streams, without using a new command, and without using in-between temporary files.<\/p>\n<p>What if we could \u2013 once and for all \u2013 leave these limitations behind and be 100% free to create <i>any<\/i> Bash command line we like, only using pipes, xargs and the Bash shell, without temporary in-between files and without starting a new command? It\u2019s possible.<\/p>\n<p>And it is not complex at all if someone shows you, but the first time this took some time and discussion to figure out. I especially want to credit and thank my previous Linux mentor, and past colleague, Andrew Dalgleish \u2013 together we figured out how to best do this just under 10 years ago.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Welcome_to_xargs_With_bash_-c\"><\/span>Welcome to xargs With bash -c<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>As we have seen, even when pipes are used in combination with xargs, one will still run into limitations for somewhat more senior engineer level scripting. Let\u2019s take our previous example and inject some debugging information without post-processing the outcome. Normally this would be hard to achieve, but not so with <code>xargs<\/code> combined with <code>bash -c<\/code>:<\/p>\n<pre>sleep 300 &amp;&#13;\npidof sleep | xargs -I{} echo \"echo 'The PID of your sleep process was: {}'; kill -9 {}; echo 'PID {} has now been terminated'\" | xargs -I{} bash -c \"{}\"&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-7988\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/1c69e5670ff7a8a66fb4922032791953\/p\/uploads\/2020\/11\/0e8181c1-1.png\" alt=\"A complex xargs command build sequence and using bash -c\" width=\"706\" height=\"128\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Here we used two <code>xargs<\/code> commands. The first one builds a custom command line, using as input the output of the previous command in the pipe (being <code>pidof sleep<\/code>) and the second xargs command executes that generated, custom-per-input (important!), command.<\/p>\n<p>Why custom-per-input? The reason is that xargs by default will process line-by-line through it\u2019s input (the output from the previous command in the pipe) and execute whatever it has been instructed to executed for each such line of input.<\/p>\n<p>There is a lot of power here. This means you can create and build any custom command and subsequently execute it, fully free of whatever format the input data is in, and fully free of having to worry about how to execute it. The only syntax you have to remember is this:<\/p>\n<pre>some_command | xargs -I{} echo \"echo '...{}...'; more_commands; more_commands_with_or_without{}\" | xargs -I{} bash -c \"{}\"&#13;\n<\/pre>\n<p>Note that the nested <code>echo<\/code> (the second echo) is only really necessary if you want to re-output actual text. Otherwise, if the second <code>echo<\/code> was not there, the first <code>echo<\/code> would start to output \u2018The PID \u2026\u2019 etc. and the <code>bash -c<\/code> subshell would be unable to parse this as a command (IOW, \u2018The PID \u2026\u2019 is not a command, and cannot be executed as such, hence the secondary\/nested echo).<\/p>\n<p>Once you remember the <code>bash -c<\/code>, the <code>-I{}<\/code> and the way to echo from within another echo (and one could alternatively use escape sequences if needbe), you will find yourself using this syntax over and over again.<\/p>\n<p>Let\u2019s say that you have to execute three things per file in the directory: 1) output the contents of the file, 2) move it to a subdirectory, 3) delete it. Normally this would require a number of steps with different staged commands, and if it gets more complex you may even need temporary files. But it is very easily done using <code>bash -c<\/code> and <code>xargs<\/code>:<\/p>\n<pre>echo '1' &gt; a&#13;\necho '2' &gt; b&#13;\necho '3' &gt; c&#13;\nmkdir subdir&#13;\nls --color=never | grep -v subdir | xargs -I{} echo \"cat {}; mv {} subdir; rm subdir\/{}\" | xargs -I{} bash -c \"{}\"&#13;\n<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-7989\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/8ce940ca85a21c7c88c6bfc91038d1b1\/p\/uploads\/2020\/11\/8a274d29-1.png\" alt=\"A fully working mini script based on xargs and bash -c in a one-liner command\" width=\"1059\" height=\"346\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Firstly quickly noting it is always a good idea to use <code>--color=never<\/code> for <code>ls<\/code>, to prevent issues on system which use color coding for directory listing outputs (on by default in Ubuntu), as this often causes strong parsing issues due to the color codes being actually sent to the terminal and prefixed to directory listing entries.<\/p>\n<p>We first create three files, a, b and c, and a subdirectory named subdir. We exclude this subdir from the directory listing with <code>grep -v<\/code> after we notice that a quick trial run (without executing the commands, or in other words without the second xargs) the subdir still shows.<\/p>\n<p>It is always important to test your complex commands first by observing the output before actually passing them to a bash subshell with <code>bash -c<\/code> for execution.<\/p>\n<p>Finally, we use the exact same method as seen previously to build or command; <code>cat<\/code> (show) the file, move the file (indicated by <code>{}<\/code>) to the subdir, and finally remove the file inside the subdir. We see that the contents of the 3 files (1, 2, 3) is being shown on screen, and if we check the current directory our files are gone. We can also see how there are no more files in the subdir. All worked well.<\/p>\n<h2 role=\"heading\" aria-level=\"2\"><span class=\"ez-toc-section\" id=\"Wrapping_up\"><\/span>Wrapping up<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Using <code>xargs<\/code> being great power to an advanced Linux (and in this case) Bash user. Using <code>xargs<\/code> in combination with <code>bash -c<\/code> brings far greater power still; the unique ability to build 100% free custom and complex command lines, without the need for intermediary files or constructs, and without the need to have stacked\/sequenced commands.<\/p>\n<p><strong>Enjoy!<\/strong>\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener noreferrer\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/7984\/using-xargs-in-combination-with-bash-c-to-create-complex-commands\/\" target=\"_blank\" rel=\"noopener noreferrer\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#Using xargs in Combination With bash -c to Create Complex Commands \u2013 CloudSavvy IT&#8221; Shutterstock\/ksb xargs is fireworks for your shell commands. Any output, generated from any shell command, can be sent to xargs for it to be processed further into another command line. Learn to harness this great power of xargs today! xargs Introduction&#8230;<\/p>\n","protected":false},"author":1,"featured_media":115584,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2020\/11\/d238ed4a.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-115583","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/115583","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=115583"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/115583\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/115584"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=115583"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=115583"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=115583"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}