Support for JSON-encoded logs in v0.6.1

Making logs easily digestible by machines is becoming a concern as tools like elasticsearch become more popular.  One of the popular strategies is to encode the whole log message in JSON and then write that as a single line to a file.  For example:

{"time": "2013-09-04T23:55:09.274041Z", "level" : "INFO", "body" : "Hello, World!" }
{"time": "2013-09-04T23:56:00.285224Z", "level" : "ERROR", "body" : "Something terrible has happened!", "tb": "  foo.c:12\n  bar.y:33" }

Unfortunately, what is good for a machine is not so great for a human.  To try to improve the situation, the latest release of lnav includes support for parsing JSON log messages and transforming them on-the-fly.  The display format is specified in a log format configuration and can specify which fields should be displayed on the main message line.  Any unused fields that are found in the message will be displayed below the main field so you don't miss anything.

The above log lines can be transformed using the following format configuration:

{
    "json_ex_log" : {
        "title" : "Example JSON Log",
        "description" : "An example log format configuration for JSON logs",
        "json" : true,
        "file-pattern" : "test-log\\.json.*",
        "level-field" : "level",
        "line-format" : [
            { "field" : "time" },
            " ",
            { "field" : "body" }
        ]
    }
}

After copying this config to ~/.lnav/formats/test/format.json, the log messages will look like this when viewed in lnav:

2013-09-04T23:55:09.274041Z Hello, World!
2013-09-04T23:56:00.285224Z Something terrible has happened!
  tb:   foo.c:12
  tb:   bar.y:33

lnav v0.6.0 released!

It's been four months since the last release of lnav  and a lot of changes have been made in the mean time.  The most exciting change, for me, is the addition of a generic log parser that will try to automatically extract data from log messages.  For example, lnav can parse the following sudo syslog message and pull out the key/value pairs for TTY, PWD, USER, and COMMAND:

Jun  3 07:00:23 Example-iMac.local sudo[2326]:    ghost : TTY=ttys002 ; PWD=/ ; USER=root ; COMMAND=/bin/ls

This requires no intervention from the user and does not require lnav to know about these log messages.  Rather, the parser searches for text that looks like "<key>=<value>" or "<key>: <value>" and makes that data available through the SQLite front-end built into lnav.  The parser can even handle some text messages if the data values are easily distinguishable from plain text.  You can read more about the parser at the new online documentation site.

Augmenting the generic log parser is the ability to define your own log formats and the addition of a few more built-in formats.  The log formats are now defined in JSON configuration files so it's much easier to add support.  The configuration files contain the regular expressions used to parse the log messages as well as the metadata that describes the values extracted from the log.  If you develop a format for a common log format please send it in for inclusion in future releases.

The user interface has also been cleaned up and should look a bit nicer.  To make learning the hotkeys a bit easier, the bottom right corner will display hints for the hotkeys to use when navigating around.  And, there's a new web site to round out the aesthetic improvements. 


EDIT: I've updated the linux binary download to v0.6.0a to fix an issue with the executable exiting immediately because the terminfo files could not be found.