PSA: logrotate does not support time-travel

栏目: IT技术 · 发布时间: 4年前

内容简介:A great many things that can break in a software system. However, diagnostics breaking the rest of the software is especially ironic. These tools are supposed to help you find bugs and other problems after the fact, not become one.The system in question wa

When diagnostics explode

A great many things that can break in a software system. However, diagnostics breaking the rest of the software is especially ironic. These tools are supposed to help you find bugs and other problems after the fact, not become one.

The system in question was a small data-recorder running on a BeagleBone Black (BBB), continiously recording measurements from specialized hardware.

These measurements are stored in an SQLite database and can be retrieved (and purged) via a very simple http interface.

For context: the BeagleBone Black is a small GNU/Linux ARM device, not unlike a Raspberry Pi.

During development, we noticed that logfiles would quickly grow to hundreds of megabyte, which could potentially be a problem if the data in the SQLite database is not retrieved, and subsequently purged, for a while. So as a precaution, we set the file-size limit to 5mb in /etc/logrotate.conf . We figured that should solve it, and during testing the logs never got very big again.

Fun in production

Imagine my surprise when I saw a 1.4gb /var/log folder that prevented any successful writes and subsequently corrupted the SQLite db. SQLite does not deal well with full disks, so this was a huge problem.

Two files especially, daemon.log and syslog, were huge with ~950mb and ~450mb respectively. They were clearly bigger than 5mb. logrotate was configured to run daily and weekly respectively. We were kind of spamming the log files, and estimated at max 50mb growth in either file per day, which should limit the files to 50mb and 350mb. But obviously, it didn’t.

Time travel

The production environment has several special properties:

  1. The BBB is not connected to the internet.
  2. There are semi-frequent power losses.
  3. The BBB does not have a battery, so power-cycling it means its internal date is reset.

What all this amounts to is: The system doesn’t know the current time and can’t get it via ntp. And whenever the system starts again, it resumes from a fixed date the disk was flashed with.

logrotate on the other hand doesn’t like that one bit. It’ll get confused by the files written in the future and even worse, it remembers when it last ran. And it doesn’t run if that’s in the future. So if the BBB runs nicely from January, 1st to July, 1st and then power-cycles, you’ll have to wait half a year for your daily logrotate run. And whenever it successfully runs, the problem will get worse.

So, in general, it’s not a good idea to run a full GNU Linux without a working clock!


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

软件开发者路线图

软件开发者路线图

Dave H. Hoover、Adewale Oshineye / 王江平 / 机械工业出版社 / 2010年9月 / 35.00元

作为一名软件开发者,你在奋力推进自己的职业生涯吗?面对今天日新月异和不断拓展的技术,取得成功需要的不仅仅是技术专长。为了增强专业性,你还需要一些软技能以及高效的学习技能。本书的全部内容都是关于如何修炼这些技能的。两位作者Dave Hoover和Adewale Oshineye给出了数十种行为模式,来帮你提高主要的技能。 本书中的模式凝结了多年的调查研究、无数次的访谈以及来自O’Reilly在......一起来看看 《软件开发者路线图》 这本书的介绍吧!

图片转BASE64编码
图片转BASE64编码

在线图片转Base64编码工具

HEX HSV 转换工具
HEX HSV 转换工具

HEX HSV 互换工具