{"id":125475,"date":"2020-12-03T16:00:57","date_gmt":"2020-12-03T13:00:57","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/parsing-html-in-bash-cloudsavvy-it\/"},"modified":"2020-12-03T16:00:57","modified_gmt":"2020-12-03T13:00:57","slug":"parsing-html-in-bash-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/parsing-html-in-bash-cloudsavvy-it\/","title":{"rendered":"#Parsing HTML in Bash \u2013 CloudSavvy IT"},"content":{"rendered":"

“#Parsing HTML in Bash \u2013 CloudSavvy IT”<\/strong><\/p>\n

\n\"Bash<\/p>\n

I have a process where I need to copy all the images from a web page. I used to run this process with xmllint<\/code>, which will process an XML or HTML file and print out the entries you specify. But when my server host provider upgraded their systems, they didn\u2019t include xmllint<\/code>. So I had to find another way to extract a list of images from an HTML page. It turns out you can do this in Bash.<\/p>\n

You may not think Bash can parse data files, but it can with some clever thinking. Bash, like other UNIX shells before it, can parse lines one at a time from a file via the built-in read<\/code> statement.<\/p>\n

By default, the read<\/code> statement scans a line of data and splits it into fields. Usually, read<\/code> splits fields using spaces and tabs, with newlines ending each line, but you can change this behavior by setting the Internal Field Separator (IFS<\/code>) value and the end-of-line delimiter (-d<\/code>).<\/p>\n

To parse an HTML file using read<\/code> , set the IFS<\/code> to a greater-than symbol (><\/code>) and the delimiter to a less-than symbol (<<\/code>). Each time Bash scans a line, it parses up to the next <<\/code> (the start of an HTML tag) then splits that data at each ><\/code> (the end of an HTML tag). This sample code takes a line of input and splits the data into the TAG<\/code> and VALUE<\/code> variables:<\/p>\n

local IFS='>'
\nread -d '<' TAG VALUE<\/pre>\n
Let\u2019s explore how this works. Consider this simple HTML file:<\/p>\n
<img src=\"https:\/\/www.cloudsavvyit.com\/8315\/parsing-html-in-bash\/logo.png\"
\nalt=\"My logo\" \/>
\n<p>some text<\/p><\/pre>\n
The first time read<\/code> parses this file, it stops at the first <<\/code> symbol. Since <<\/code> is the first character of this sample input, that means Bash finds an empty string. The resulting TAG<\/code> and VALUE<\/code> strings are also empty. But that\u2019s fine for my use case.<\/p>\n
The next time Bash reads the input, it gets img src=\"https:\/\/www.cloudsavvyit.com\/8315\/parsing-html-in-bash\/logo.png\"\u21b2alt=\"My logo\" \/>\u21b2<\/code> with a newline right before the alt, and stops before the\u00a0<<\/code> symbol on the next line. Then read<\/code> splits the line at the ><\/code> symbol, which leaves TAG<\/code> with img src=\"https:\/\/www.cloudsavvyit.com\/8315\/parsing-html-in-bash\/logo.png\"\u21b2alt=\"My logo\" \/<\/code> and VALUE<\/code> with an empty newline.<\/p>\n
The third time read<\/code> parses the HTML file, it gets\u00a0p>some text<\/code>. Bash splits the string at the ><\/code> resulting in TAG<\/code> containing p<\/code> and VALUE<\/code> with some text<\/code> .<\/p>\n
Now that you understand how to use read<\/code>, it\u2019s easy to parse a longer HTML file with Bash. Start with a Bash function called xmlgetnext<\/code> to parse the data using read<\/code> , since you\u2019ll be doing this again and again in the script<\/a>. I named my function xmlgetnext<\/code> to remind me this is a replacement for the Linux xmllint<\/code> program, but I could have just as easily named it htmlgetnext<\/code> .<\/p>\n
xmlgetnext () {
\nlocal IFS='>'
\nread -d '<' TAG VALUE
\n}<\/pre>\n
Now call that xmlgetnext<\/code> function to parse the HTML file. This is my complete htmltags<\/code> script:<\/p>\n
#!\/bin\/sh
\n# print a list of all html tags
\n
\nxmlgetnext () {
\nlocal IFS='>'
\nread -d '<' TAG VALUE
\n}
\n
\ncat $1 | while xmlgetnext ; do echo $TAG ; done<\/pre>\n
The last line is the key. It loops through the file using xmlgetnext<\/code> to parse the HTML, and prints out only the TAG<\/code> entries. And because of how echo<\/code> operates with the standard field separators, any lines like img src=\"https:\/\/www.cloudsavvyit.com\/8315\/parsing-html-in-bash\/logo.png\"\u21b2alt=\"My logo\" \/<\/code> that contain a newline get printed on a single line, as img src=\"https:\/\/www.cloudsavvyit.com\/8315\/parsing-html-in-bash\/logo.png\" alt=\"My logo\" \/<\/code>.<\/p>\n
\"Screenshot
Parsing HTML in Bash<\/figcaption><\/figure>\n
To fetch just the list of images, I run the output of this script through grep<\/code>\u00a0to only print the lines that have an img<\/code> tag at the start of the line.\n<\/div>\n
If you liked the article, do not forget to share it with your friends. Follow us on\u00a0Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n
\n
For forums sites go to Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n
\n
If you want to read more like this article, you can visit our Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n
Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
“#Parsing HTML in Bash \u2013 CloudSavvy IT” I have a process where I need to copy all the images from a web page. I used to run this process with xmllint, which will process an XML or HTML file and print out the entries you specify. But when my server host provider upgraded their systems,…<\/p>\n","protected":false},"author":1,"featured_media":125476,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2017\/07\/add8ac45.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-125475","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/125475","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=125475"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/125475\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/125476"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=125475"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=125475"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=125475"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}