{"id":83676,"date":"2020-10-07T11:00:59","date_gmt":"2020-10-07T08:00:59","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it\/"},"modified":"2020-10-07T11:00:59","modified_gmt":"2020-10-07T08:00:59","slug":"what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it\/","title":{"rendered":"#What is a Log Management Tool, and Should You Use One? \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a303549d1e07\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a303549d1e07\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it\/#Why_Should_I_Care_About_Log_Files\" >Why Should I Care About Log Files?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it\/#The_Problem_Too_Many_Servers_Too_Many_Logs\" >The Problem: Too Many Servers, Too Many Logs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it\/#How_Do_Log_Management_Tools_Work\" >How Do Log Management Tools Work?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/what-is-a-log-management-tool-and-should-you-use-one-cloudsavvy-it\/#What_Tool_Should_I_Use\" >What Tool Should I Use?<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#What is a Log Management Tool, and Should You Use One? \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<figure id=\"attachment_421\" style=\"width: 700px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-421 size-full\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/262abeb53fef592183c26d800a2eaafc\/p\/uploads\/2019\/06\/403dfd6c.png\" alt=\"\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><figcaption class=\"wp-caption-text\"><span class=\"imagecredit\"><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/kibana\">Kibana<\/a><\/span><\/figcaption><\/figure>\n<p>Log files contain a lot of important data about how your infrastructure is performing, but when they\u2019re thousands of lines long, gaining useful insights from them can be hard. Log management tools help fix this problem.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Why_Should_I_Care_About_Log_Files\"><\/span>Why Should I Care About Log Files?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Every connection to your web server is logged; anytime a user requests a resource, a line in the log file is written. You can use these kinds of logs to get a very accurate view of traffic coming in to your site. They doesn\u2019t offer any data about how the user interacts with the site, which is left to analytics tools, but they do tell you how your web server is handling each request.<\/p>\n<p>The HTTP status code of the request is often logged, so these logs can be useful for tracking down broken links and errors that return 404 (which can affect your site\u2019s ranking when search engines like Google crawl the site), something that isn\u2019t returned with most analytics tools (as your page isn\u2019t even loaded).<\/p>\n<p><a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">App<\/a>lications create logs of errors, which is useful for tracking down problems in the backend. If a particular API is causing errors, it will pop up in the log files very quickly. Your own applications will require you to implement your own logs, but there are plenty of <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/github.com\/winstonjs\/winston\">logging libraries<\/a> that make the process easier.<\/p>\n<p>Unix keeps logs of everything that goes on with the system. Every command you enter is logged to <code>~\/.bash_history<\/code>, every login attempt (including failed, possibly malicious ones) are logged to <code>\/var\/log\/auth.log<\/code>, and most other system events will generate their own log files, usually stored in <code>\/var\/log\/<\/code>.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Problem_Too_Many_Servers_Too_Many_Logs\"><\/span>The Problem: Too Many Servers, Too Many Logs<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-200\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/5c64c44858f0398fa6aeb9a27f846f1c\/p\/uploads\/2019\/05\/65436fe7-3.png\" alt=\"\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Most applications leave behind logs, a paper trail of what that application has been doing. Some applications, like web servers, can leave behind a lot of logs, which can get large enough to fill up your server\u2019s hard drive and need regular rotating.<\/p>\n<p>One server is hard enough to manage, but managing logs spread across multiple servers can become an impossible task, requiring you to authenticate on each server and manually view the log files for that particular machine.<\/p>\n<p>Log management tools are the solution to this problem, allowing you to concentrate your logs in one place and view them much more easily. Many services provide also visualization tools, so you won\u2019t have to go digging through ten thousand lines of text to find useful data.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"How_Do_Log_Management_Tools_Work\"><\/span>How Do Log Management Tools Work?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-413\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/de8619f141a050b9496b7a14a475ffdb\/p\/uploads\/2019\/06\/ea691263.png\" alt=\"\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>A log management tool like <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/docs.fluentd.org\">Fluentd<\/a> will run on a server somewhere, whether that\u2019s in the cloud behind a managed web interface or self hosted on your own systems. The server that this runs on is called an aggregator server, and collects logs from multiple external sources.<\/p>\n<p>The process starts with ingest\u2014log files from client system are fed into the aggregator with the help of a program called a log shipper. Log shippers like the <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.rsyslog.com\/\"><code>rsyslog<\/code><\/a>\u00a0library are lightweight libraries that sit on client systems, and point towards the aggregate server.<\/p>\n<p>Once the log files are ingested, what happens to them is up to the log management tool. For some tools, the simple collection of them is enough, and they can be sorted and fed into a time <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/watch-movies-tv-seriess\/\" data-internallinksmanager029f6b8e52c=\"8\" title=\"Watch Movies &amp; TV Series\" target=\"_blank\" rel=\"noopener\">series<\/a> database like <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.influxdata.com\/products\/influxdb-overview\/influxdb-2-0\/\">InfluxDB<\/a> for further analysis. For others, like <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.graylog.org\/\">Graylog<\/a>, the service is built around the quality of their visualization and analytics tools.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"What_Tool_Should_I_Use\"><\/span>What Tool Should I Use?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-415\" src=\"https:\/\/www.cloudsavvyit.com\/thumbcache\/0\/0\/347d94590a538700f43acb98bdbcde03\/p\/uploads\/2019\/06\/9eb79c67.png\" alt=\"\" width=\"700\" height=\"300\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/\">The Elastic Stack<\/a>\u00a0(also called the ELK stack) is a very popular logging platform. It\u2019s comprised of four different applications, all open source with the same developers. It\u2019s entirely free, but you\u2019ll need to host it yourself.<\/p>\n<ul>\n<li><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/beats\">Beats<\/a>\u00a0are lightweight log shippers designed to be installed on client machines, and send data to the other applications in the stack.<\/li>\n<li><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/logstash\">Logstash<\/a>\u00a0is the ingestion engine, which can take data from Beats or other programs such as <code>rsyslog<\/code>\u00a0and prepare it to be sent off to Elasticsearch (or another analytics engine).<\/li>\n<li><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/elasticsearch\">Elasticsearch<\/a>\u00a0is the engine at the center of the Elastic stack (after which the stack is named). It functions as a database for storing your logs (and other objects) and exposes a RESTful API for use in other applications.<\/li>\n<li><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/kibana\">Kibana<\/a>\u00a0is the frontend for the Elastic Stack, and provides all of the visualization, charts, graphs, and search options for the end user.<\/li>\n<\/ul>\n<p>Many of the tools in the Elastic Stack are fairly plug and play with other log management tools, so if you have a preference for something else, you can likely replace that item in the stack. Overall though, most tools and frameworks will follow the same <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/general\/\" data-internallinksmanager029f6b8e52c=\"3\" title=\"General\" target=\"_blank\" rel=\"noopener\">general<\/a> structure as the Elastic Stack\u2014log shipper &gt; ingestion engine &gt; database &gt; visualization tool.<\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.fluentd.org\/\">Fluentd<\/a>\u00a0and <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.elastic.co\/products\/beats\/filebeat\">Filebeat<\/a> are alternative ingestion engines, and would replace Logstash in the stack. These can feed data into a time series database like <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.influxdata.com\/products\/influxdb-overview\/influxdb-2-0\/\">InfluxDB<\/a>, which has a built-in plugin for <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/grafana.com\/grafana?feature=DS_InfluxDB&amp;from=homepage#datasources\">Grafana<\/a>, an analytics and visualization platform.<\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-install-and-use-logwatch-log-analyzer-and-reporter-on-a-vps\">Logwatch<\/a> is a very basic command line utility that monitors your log files and sends you a daily report. It doesn\u2019t do any kind of collection, so it\u2019s ideal for single server setups that want some more insight into their server\u2019s logs.<\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.graylog.org\/\">Graylog<\/a> replaces the Elastic Stack completely, and only requires external log shippers to ingest data. Their web interface supports creating custom charts and dashboards for monitoring your logs, but may be lacking compared to a setup with a proper database and Grafana.<\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/papertrailapp.com\/\">SolarWinds Papertrail<\/a>\u00a0is a fully managed service that displays logs in realtime, which can be very useful when debugging issues with your servers. Their plans are <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/papertrailapp.com\/plans\">fairly cheap<\/a>, being segmented per GB and starting at just $7.<\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.splunk.com\">Splunk<\/a> monitors just about everything surrounding your applications, including logs. If you want a comprehensive analytics suite, Splunk may be for you.<\/p>\n<p><a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/logdna.com\/pricing\/\">LogDNA<\/a>\u00a0is a simple log analysis tool with <a rel=\"nofollow noopener noreferrer\" target=\"_blank\" href=\"https:\/\/logdna.com\/pricing\">very cheap plans<\/a>. If you\u2019re looking for an easy alternative to configuring an ELK stack, LogDNA can be set up quickly.\n<\/div>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener noreferrer\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/400\/what-is-a-log-management-tool-and-should-you-use-one\/\" target=\"_blank\" rel=\"noopener noreferrer\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#What is a Log Management Tool, and Should You Use One? \u2013 CloudSavvy IT&#8221; Kibana Log files contain a lot of important data about how your infrastructure is performing, but when they\u2019re thousands of lines long, gaining useful insights from them can be hard. Log management tools help fix this problem. Why Should I Care&#8230;<\/p>\n","protected":false},"author":1,"featured_media":83677,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2019\/06\/403dfd6c.png","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-83676","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/83676","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=83676"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/83676\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/83677"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=83676"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=83676"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=83676"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}