Shipping Multiline Logs with Filebeat

栏目: IT技术 · 发布时间: 4年前

内容简介:Multiline logs provide valuable information for developers when troubleshooting issues with applications. An example of this is theSo, the stack trace above would be seen in Kibana as four separate documents. This makes it difficult to search and understan
Shipping Multiline Logs with Filebeat

Multiline logs provide valuable information for developers when troubleshooting issues with applications. An example of this is the stack trace . A stack trace is a sequence of method calls that an application was in the middle of when an exception was thrown. The stack trace includes the line in question that encountered the error, as well as the error itself. An example of a Java stack trace can be seen here: 

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

When using a logging tool like the ELK stack, it can be difficult to identify and search for a stack trace without the right configuration in place. When shipping application logs with an open source light shipper like Filebeat, each line of a stack trace will be seen in Kibana as an individual document.

So, the stack trace above would be seen in Kibana as four separate documents. This makes it difficult to search and understand errors and exceptions within the stack trace as they are divorced from their context as a common event. When logging application logs with Filebeat, users can avoid this issue by adding configuration options in the filebeat.yml file.

You can configure the filebeat.yml input section filebeat.inputs to  add a few multiline configuration options to make sure that multiline logs, like stack traces, are sent as one complete document. Adding the configuration options below to the filebeat.yml input section will ensure that the Java stack trace referenced above will be sent as a single document. 

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
Shipping Multiline Logs with Filebeat

Multiline logging in Filebeat

mutliline.pattern – This configuration option defines the regular expression pattern to match. In the example above the regular expression is matching any line that begins with whitespace up to the previous line.

multiline.negate – This option defines if the pattern is negated. The default is false.

multiline.match – This option determines how Filebeat combines matching lines into an event. This option depends on the value for negate . In the example above, we set negate to false and match to after. This means that consecutive lines that match the pattern are attached to the previous line that does not match the pattern. 

Along with the multiline configuration options mentioned above, you can set options to flush the memory of a multiline message, set the maximum number of lines that can be included in a single event, and you can increase the timeout, which is set to 5 seconds by default.

Let’s take a look at an example using the multiline.flush_pattern. This configuration option with Filebeat is useful for multiline application logs that contain events that start and end with specific markers.

[2015-08-24 11:49:14,389] Start new event
[2015-08-24 11:49:14,395] Content of processing something
[2015-08-24 11:49:14,399] End event

If we want these lines to be seen in Kibana as a single document we would use the following multiline configuration options in the filebeat.yml :

multiline.pattern: ‘Start new event’
multiline.negate: true
multiline.match: after
multiline.flush_pattern: ‘End event’

From the configuration options above, when the pattern “Start new event” is seen and the following lines do not match the pattern, they will be appended to the previous line that does match the pattern. The flush_pattern option will then signal that the multiline event is over when a line is seen beginning with “End event.”

Shipping Multiline Logs with Filebeat

Multiline logging in Filebeat

Conclusion

Centralizing your application logs into a single location is an important first step to help troubleshoot any issues that arise with your application. Making sure that your logs are being ingested and displayed correctly within that tool can help companies decrease their mean time to resolution.


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

互联网思维独孤九剑

互联网思维独孤九剑

赵大伟 / 机械工业出版社 / 2014-3-20 / 49

《互联网思维独孤九剑》是国内第一部系统阐述互联网思维的著作,用9大互联网思维:用户思维、简约思维、极致思维、迭代思维、流量思维、社会化思维、大数据思维、平台思维、跨界思维,以专业的视角全方位解读移动互联网给传统产业带来的变革,涉及战略规划、商业模式设计、品牌建设、产品研发、营销推广、组织转型、文化变革等企业经营价值链条的各个方面。这是一部传统企业互联网转型必读的“孙子兵法”,帮助我们开启对新商业文......一起来看看 《互联网思维独孤九剑》 这本书的介绍吧!

CSS 压缩/解压工具
CSS 压缩/解压工具

在线压缩/解压 CSS 代码

XML、JSON 在线转换
XML、JSON 在线转换

在线XML、JSON转换工具

RGB CMYK 转换工具
RGB CMYK 转换工具

RGB CMYK 互转工具