Problem
You want to merge a collection of log files into a single, chronological log file.
Solution
#!/bin/sh
perl -ne
'print $last, /last message repeated d+ times$/ ? "" : "n" if $last;
chomp($last = $_);
if (eof) {
print;
undef $last;
}' "$@" | sort -s -k 1,1M -k 2,2n -k 3,3 | tr '' 'n'
Discussion
The system logger automatically prepends a timestamp to each message, like this:
Feb 21 12:34:56 buster kernel: device eth0 entered promiscuous mode
To merge log files, sort each one by its timestamp entries, using the first three fields (month, date, and time) as keys.
A complication arises because the system logger inserts “repetition messages” to conserve log file space:
Feb 21 12:48:16 buster last message repeated 7923 times
The timestamp for the repetition message is often later than the last message. It would be terribly misleading if possibly unrelated messages from other log files were merged between the last message and its associated repetition message.
To avoid this, our Perl script glues together the last message with a subsequent repetition message (if present), inserting a null character between them: this is reliable because the system logger never writes null characters to log files. The script writes out the final line before the end of each file and then forgets the last line, to avoid any possibility of confusion if the next file happens to start with an unrelated repetition message.
The sort command sees these null-glued combinations as single lines, and keeps them together as the files are merged. The null characters are translated back to newlines after the files are sorted, to split the combinations back into separate lines.
We use sort -s to avoid sorting entire lines if all of the keys are equal: this preserves the original order of messages with the same timestamp, at least within each original log file.
If you have configured the system logger to write messages to multiple log files, then you may wish to remove duplicates as you merge. This can be done by using sort -u instead of -s, and adding an extra sort key -k 4 to compare the message contents. There is a drawback, however: messages could be rearranged if they have the same timestamp. All of the issues related to sort -s and -u are consequences of the one-second resolution of the timestamps used by the system logger.
We’ll note a few other pitfalls related to timestamps. The system logger does not record the year, so if your log files cross a year boundary, then you will need to merge the log files for each year separately, and concatenate the results. Similarly, the system logger writes timestamps using the local time zone, so you should avoid merging log files that cross a daylight saving time boundary, when the timestamps can go backward. Again, split the log files on either side of the discontinuity, merge separately, and then concatenate.
If your system logger is configured to receive messages from other machines, note that the timestamps are generated on the machine where the log files are stored. This allows consistent sorting of messages even from machines in different time zones.
See Also
sort(1).
*Reprinted with the permission of the O’Reilly Network.
SIgn up here for FREE technology newsletters from murdok!
The Linux Security Cookbook includes real solutions to a wide range of targeted problems, such as sending encrypted email within Emacs, restricting access to network services at particular times of day, firewalling a webserver, preventing IP spoofing, setting up key-based SSH authentication, and much more. With over 150 ready-to-use scripts and configuration files, this unique book helps administrators secure their systems without having to look up specific syntax.
