Shipping Multiline Logs with Filebeat

栏目: IT技术 · 发布时间: 4年前

内容简介:Multiline logs provide valuable information for developers when troubleshooting issues with applications. An example of this is theSo, the stack trace above would be seen in Kibana as four separate documents. This makes it difficult to search and understan
Shipping Multiline Logs with Filebeat

Multiline logs provide valuable information for developers when troubleshooting issues with applications. An example of this is the stack trace . A stack trace is a sequence of method calls that an application was in the middle of when an exception was thrown. The stack trace includes the line in question that encountered the error, as well as the error itself. An example of a Java stack trace can be seen here: 

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

When using a logging tool like the ELK stack, it can be difficult to identify and search for a stack trace without the right configuration in place. When shipping application logs with an open source light shipper like Filebeat, each line of a stack trace will be seen in Kibana as an individual document.

So, the stack trace above would be seen in Kibana as four separate documents. This makes it difficult to search and understand errors and exceptions within the stack trace as they are divorced from their context as a common event. When logging application logs with Filebeat, users can avoid this issue by adding configuration options in the filebeat.yml file.

You can configure the filebeat.yml input section filebeat.inputs to  add a few multiline configuration options to make sure that multiline logs, like stack traces, are sent as one complete document. Adding the configuration options below to the filebeat.yml input section will ensure that the Java stack trace referenced above will be sent as a single document. 

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
Shipping Multiline Logs with Filebeat

Multiline logging in Filebeat

mutliline.pattern – This configuration option defines the regular expression pattern to match. In the example above the regular expression is matching any line that begins with whitespace up to the previous line.

multiline.negate – This option defines if the pattern is negated. The default is false.

multiline.match – This option determines how Filebeat combines matching lines into an event. This option depends on the value for negate . In the example above, we set negate to false and match to after. This means that consecutive lines that match the pattern are attached to the previous line that does not match the pattern. 

Along with the multiline configuration options mentioned above, you can set options to flush the memory of a multiline message, set the maximum number of lines that can be included in a single event, and you can increase the timeout, which is set to 5 seconds by default.

Let’s take a look at an example using the multiline.flush_pattern. This configuration option with Filebeat is useful for multiline application logs that contain events that start and end with specific markers.

[2015-08-24 11:49:14,389] Start new event
[2015-08-24 11:49:14,395] Content of processing something
[2015-08-24 11:49:14,399] End event

If we want these lines to be seen in Kibana as a single document we would use the following multiline configuration options in the filebeat.yml :

multiline.pattern: ‘Start new event’
multiline.negate: true
multiline.match: after
multiline.flush_pattern: ‘End event’

From the configuration options above, when the pattern “Start new event” is seen and the following lines do not match the pattern, they will be appended to the previous line that does match the pattern. The flush_pattern option will then signal that the multiline event is over when a line is seen beginning with “End event.”

Shipping Multiline Logs with Filebeat

Multiline logging in Filebeat

Conclusion

Centralizing your application logs into a single location is an important first step to help troubleshoot any issues that arise with your application. Making sure that your logs are being ingested and displayed correctly within that tool can help companies decrease their mean time to resolution.


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

The Joy of X

The Joy of X

Niall Mansfield / UIT Cambridge Ltd. / 2010-7-1 / USD 14.95

Aimed at those new to the system seeking an overall understanding first, and written in a clear, uncomplicated style, this reprint of the much-cited 1993 classic describes the standard windowing syste......一起来看看 《The Joy of X》 这本书的介绍吧!

HTML 编码/解码
HTML 编码/解码

HTML 编码/解码

XML 在线格式化
XML 在线格式化

在线 XML 格式化压缩工具

RGB CMYK 转换工具
RGB CMYK 转换工具

RGB CMYK 互转工具