PSA: logrotate does not support time-travel

栏目: IT技术 · 发布时间: 4年前

内容简介:A great many things that can break in a software system. However, diagnostics breaking the rest of the software is especially ironic. These tools are supposed to help you find bugs and other problems after the fact, not become one.The system in question wa

When diagnostics explode

A great many things that can break in a software system. However, diagnostics breaking the rest of the software is especially ironic. These tools are supposed to help you find bugs and other problems after the fact, not become one.

The system in question was a small data-recorder running on a BeagleBone Black (BBB), continiously recording measurements from specialized hardware.

These measurements are stored in an SQLite database and can be retrieved (and purged) via a very simple http interface.

For context: the BeagleBone Black is a small GNU/Linux ARM device, not unlike a Raspberry Pi.

During development, we noticed that logfiles would quickly grow to hundreds of megabyte, which could potentially be a problem if the data in the SQLite database is not retrieved, and subsequently purged, for a while. So as a precaution, we set the file-size limit to 5mb in /etc/logrotate.conf . We figured that should solve it, and during testing the logs never got very big again.

Fun in production

Imagine my surprise when I saw a 1.4gb /var/log folder that prevented any successful writes and subsequently corrupted the SQLite db. SQLite does not deal well with full disks, so this was a huge problem.

Two files especially, daemon.log and syslog, were huge with ~950mb and ~450mb respectively. They were clearly bigger than 5mb. logrotate was configured to run daily and weekly respectively. We were kind of spamming the log files, and estimated at max 50mb growth in either file per day, which should limit the files to 50mb and 350mb. But obviously, it didn’t.

Time travel

The production environment has several special properties:

  1. The BBB is not connected to the internet.
  2. There are semi-frequent power losses.
  3. The BBB does not have a battery, so power-cycling it means its internal date is reset.

What all this amounts to is: The system doesn’t know the current time and can’t get it via ntp. And whenever the system starts again, it resumes from a fixed date the disk was flashed with.

logrotate on the other hand doesn’t like that one bit. It’ll get confused by the files written in the future and even worse, it remembers when it last ran. And it doesn’t run if that’s in the future. So if the BBB runs nicely from January, 1st to July, 1st and then power-cycles, you’ll have to wait half a year for your daily logrotate run. And whenever it successfully runs, the problem will get worse.

So, in general, it’s not a good idea to run a full GNU Linux without a working clock!


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

Effective Python

Effective Python

布雷特·斯拉特金(Brett Slatkin) / 爱飞翔 / 机械工业出版社 / 2016-1 / 59

用Python编写程序,是相当容易的,所以这门语言非常流行。但若想掌握Python所特有的优势、魅力和表达能力,则相当困难,而且语言中还有很多隐藏的陷阱,容易令开发者犯错。 本书可以帮你掌握真正的Pythonic编程方式,令你能够完全发挥出Python语言的强大功能,并写出健壮而高效的代码。Scott Meyers在畅销书《Effective C++》中开创了一种以使用场景为主导的精练教学方......一起来看看 《Effective Python》 这本书的介绍吧!

JS 压缩/解压工具
JS 压缩/解压工具

在线压缩/解压 JS 代码

JSON 在线解析
JSON 在线解析

在线 JSON 格式化工具

正则表达式在线测试
正则表达式在线测试

正则表达式在线测试