lnav v0.8.1 released!

After way too long, lnav v0.8.1 has been released.  Here is a rundown of the changes:

Features:
     * Added a spectrogram command and view that displays the values of a
       numeric field over time.  The view works for log message fields or
       for database result columns.
     * Log formats can now create SQL views and execute other statements
       by adding '.sql' files to their format directories.  The SQL scripts
       will be executed on startup.
     * Added 'json_group_object' and 'json_group_array' aggregate SQL
       functions that collects values from a GROUP BY query into a JSON
       object or array, respectively.
     * The SQL view will now graph values found in JSON objects/arrays in
       addition to the regular columns in the result.
     * Added an 'regexp_match(<re>, <str>)' SQL function that can be used to
       extract values from a string using a regular expression.
     * Added an 'extract(<str>)' SQL function that extracts values using the
       same data discover/extraction parser used in the 'logline' table.
     * Added a "summary" overlay line to the bottom of the log view that
       displays how long ago the last message was received, along with the
       total number of files and the error rate over the past five minutes.
     * Pressing 'V' in the DB view will now check for a column with a
       timestamp and move to the corresponding time in the log view.
     * Added 'a/A' hotkeys to restore a view previously popped with 'q/Q'.
     * Added ":hide-lines-before", ":hide-lines-after", and
       ":show-lines-before-and-after" commands so that you can filter out
       log lines based on time.
     * Scripts containing lnav commands/queries can now be executed using
       the pipe ('|') hotkey.  See the documentation for more information.
     * Added an ":eval" command that can be used to execute a command or
       query after performing environment variable substitution.
     * Added an ":echo" command that can be useful for scripts to message
       the user.
     * The "log_part" column can now be set with an SQL UPDATE statement.
     * Added a "log_body" hidden column that returns the body of the log
       message.
     * Added ":config", ":reset-config", and ":save-config" commands to change
       configuration options, reset to default, and save them for future
       executions.
     * Added a "/ui/clock-format" configuration option that controls the time
       format in the top-left corner.
     * Added a "/ui/dim-text" configuration option that controls the brightness
       of text in the UI.
     * Added support for TAI64 timestamps (http://cr.yp.to/libtai/tai64.html).
     * Added a safe execution mode. If the 'LNAVSECURE' environment variable is
       set before executing lnav, the following commands are disabled:
       - open
       - pipe-to
       - pipe-line-to
       - write-*-to
       This makes it easier to run lnav with escalated privileges in restricted
       environments, without the risk of users being able to use the above
       mentioned commands to gain privileged access.

     Interface Changes:
     * The 'o/O' hotkeys have been reassigned to navigate through log
       messages that have a matching "opid" field.  The old action of
       moving forward and backward by 60 minutes can be simulated by
       using the ':goto' command with a relative time and the 'r/R'
       hotkeys.
     * Log messages with timestamps that pre-date previous log messages will
       have the timestamp highlighted in yellow and underlined.  These out-
       of-time-order messages will be assigned the time of the previous
       message for sorting purposes.  You can press the 'p' hotkey to examine
       the 'Received Time' of the message as well as the time parsed from the
       original message.  A "log_actual_time" hidden field has also been
       added to the SQLite virtual table so you can operate on the original
       message time from the file.
     * The 'A/B' hotkeys for moving forward/backward by 10% line increments
       have been reassigned to '[' and ']'.  The 'a' and 'A' hotkeys are now
       used to return to the previously popped view while trying to preserve
       the time range.  For example, after leaving the spectrogram view with
       'q', you can press 'A' return to the view with the top time in the
       spectrogram matching the top time in the log view.
     * The 'Q' hotkey now pops the current view off of the stack while
       maintaining the top time between views.

     Fixes:
     * Issues with tailing JSON logs have been fixed.
     * The jget() SQL function should now work for objects nested in arrays.

Terminal.app in OS X 10.11 (El Capitan) has mouse support!

Basic mouse support has been in lnav for a little while now, but OS X's Terminal.app never had support for it, so I never used it much.  Fortunately, with the release of El Capitan, mouse support has been added and it seems to work well enough for lnav.

To toggle mouse support on and off, press F2 in lnav.  You can then click and drag to mark lines in the log or drag the scroll bar on the right side.  The gif below shows the functionality in action.

lnav v0.8.0 released!

This release of lnav adds quite a few new features and fixes some nagging bugs.

Features:

  • Integration with "papertrailapp.com" for querying and tailing server log and syslog messages. See the Papertrail section in the online help for more details.
  • Remote files can be opened when lnav is built with libcurl v7.23.0+
  • SQL queries can now be done on lines that match a regular expression using the "log_search" table or by creating custom tables with the ":create-search-table" command.
  • Log formats that are "containers" for other log formats, like syslog, are now supported. See the online help for more information.
  • Formats can be installed from git repositories using the '-i' option. A standard set of extra formats can be installed by doing 'lnav -i extra'. (You must have git installed for this to work.)
  • Added support for 'VMware vSphere Auto Deploy' log format.
  • Added a 'sudo' log format.
  • Added hotkeys to move left/right by a smaller increment (H/L or Shift+Left/Shift+Right).
  • A color-coded bar has been added to the left side to show where messages from one file stop and messages from another file start.
  • The '-C' option will now try to check any specified log files to make sure the format(s) match all of the lines.
  • Added an "all_logs" SQLite table that contains the message format extracted from each log line. Also added a ';.msgformat' SQL command that executes a query that returns the counts for each format and the first line where the format was seen.
  • Added an "lnav_views" SQLite table that can be used to query and change the lnav view state.
  • When typing in a command, the status bar will display a short summary of the currently entered command.
  • Added a "delete-filter" command.
  • Added a "log_msg_instance" column to the logline and log_search tables to make it easier to join tables that are matching log messages that are ordered.
  • Added a "timeslice()" function to SQLite so that it is easier to group log messages by time buckets.
  • The 'goto' command now supports relative time values like 'a minute ago', 'an hour later', and many more.

Interface Changes:

  • The 'r/R' hotkeys have been reassigned to navigate through the log messages by the relative time value that was last used with the 'goto' command.

Fixes:

  • The pretty-print view should now work for text files.
  • Nested fields in JSON logs are now supported for levels, bodies, etc...
  • Tab-completion should work for quoted SQL identifiers.
  • 'lo-fi' mode key shortcut changed to CTRL+L.
  • 'redraw' shortcut removed. Relegated to just a command.
  • Fixed lnav hang in pretty-print mode while doing a dns lookup.
  • The generic log message parser used to extract data has been optimized and should be a bit faster.

Pretty-print view in v0.7.3

I wanted to call out the pretty-print feature in the latest release of lnav.  This idea came from a coworker of Suresh who was having a hard time trying to read some unformatted XML in a log.  They wanted the XML pretty-printed and were hoping that could be done by just piping the message to xmlpp or the like.  So, first we implemented the 'pipe-to' and 'pipe-line-to' commands that will let you pipe log messages to a command and then display the result inside of lnav.  That worked well enough, but pretty-printing is such a frequent operation that having to execute a command was kind of a pain.  It would also be nice if it worked for a variety of text, like JSON or Python data.  The solution we came up with was to leverage the existing code for parsing log messages to create a simple pretty-printer that should work for most data formats.  Another benefit is that the log message does not have to be well-formed for the printer to work, any leading or trailing garbage shouldn't confuse things.

As an example, here is a screenshot of the log message with the unformatted XML text with word-wrapping turned on:

That's not very easy to read and it's hard to figure out the structure of the message.  Now, here is that same message after pressing SHIFT+P to switch to the pretty-print view of lnav:

The XML text is indented nicely and the usual syntax highlighting is applied.  Also notice that lnav will automatically try to lookup the DNS name for IP addresses.  Overall, I think it's a major improvement over the raw view.

This is a pretty simple feature but I have found it quite useful in the couple weeks that it has been implemented.  It's so useful that I'm kicking myself for not having thought of it before.

lnav v0.7.3 released!

This release of lnav has a few new features, like the pretty-print view, and fleshes out the list of commands.  Here is the full listing of what has changed:

     Features:
     * Add 'pipe-to' and 'pipe-line-to' commands that pipe the currently
       marked lines or the current log message to a shell command,
       respectively.
     * Added a "pretty-print" view (P hotkey) that tries to reformat log
       messages so that they are easier to read.
     * Added a 'redraw' command (CTRL+L hotkey) to redraw the window in
       case it has been corrupted.
     * Added a 'relative-goto' command to move the current view relative
       to its current position.
     * Experimental support for linking with jemalloc.
     * The plain text view now supports filtering.
     * Added 'next-mark' and 'prev-mark' commands to jump to the next or
       previous bookmarked line (e.g. error, warning, ...)
     * Added a 'zoom-to' command to change the zoom level of the histogram
       view.
     * Log formats can now define their own timestamp formats with the
       'timestamp-format' field.

     Fixes:
     * Autotools scripts overhaul.
     * Added a configure option to disable linking with libtinfo. The newer
       versions of ncurses don't require it, however the build silently pulls
       it in as a dependency, if it is available on the system. This can be
       explicitly disabled using the '--disable-tinfo' option during configure.
     * Fixed the configure script behavior to ignore the values specified using
       the CFLAGS and LDFLAGS environment variables while searching for sqlite3
       when '--with-sqlite3' switch was specified without the prefix.
     * The configure script now recognizes libeditline symlink'ed to masquerade
       as libreadline. This previously used to cause problems at compile time,
       specially on OS X. If you come across this error, use the
       '--with-readline=prefix' switch to specify the path to the correct
       location of libreadline.
     * The order that log formats are tried against a log file is now
       automatically determined so that more specific formats are tested
       before more general ones.  The order is determined on startup based on
       how each format matches each other formats sample lines.
     * Command files (i.e. those executed via the '-f' flag) now support
       commands/queries that span more than one line.
     * Added more log levels: stats, debug2 - debug5.

You can check out this release on GitHub to get more information and the links for downloads.

lnav v0.7.2 released!

This maintenance release of lnav fixes a few bugs and adds the following features:

     * Added log formats for vdsm, openstack, and the vmkernel.

     * Added a "lo-fi" mode (L hotkey) that dumps the displayed log lines to the terminal without any decorations.  The write-to, write-json-to, and write-csv-to commands will also write their output to the terminal when passed '-' as the file name.  This mode can be useful for copying plain text lines to the clipboard.

     * (OS X) Text search strings are copied to the system's "find" clipboard. Also, when starting a new search, the current value in the "find" clipboard can be tab-completed.

You can check out this release on GitHub to get more information and the links for downloads.

 

lnav v0.7.1 released!

After a long hiatus there's a new release of lnav with a few new features and bug fixes.  Here's what is new:

  • Added an 'environ' SQL table that reflects lnav's environment variables.  The table can be read and written to using SQL SELECT, INSERT, UPDATE, and DELETE statements.  Setting variables can be a way to use SQL query results in lnav commands.

  • Added a 'jget' SQLite function that can extract fields from a JSON-encoded value.

  • Added log formats for the OpenAM identity provider.

  • Added a 'clear-highlight' command to clear previous calls to the 'highlight' command.

  • Fixed some performance bugs in indexing JSON log formats.  Loading times should be at least five times faster.

  • Filtering performance should be improved so that enabling/disabling filters should be almost instantaneous.

  • The filter-in, filter-out, and highlight commands now support tab-completion of text in the current view.

  • Added an '-i' flag that installs format files in: ~/.lnav/formats/installed

You can check out this release on GitHub to get more information and links to the available downloads.

lnav v0.7.0 released!

The latest release of lnav is out and features quite a few improvements and bug fixes.  Here is a quick rundown of what is new:

  • Added a '.schema' SQL command to open a view that displays the schema for the internal tables and any attached databases. If lnav was only executed with a SQLite database and no text files, this view will open by default.
  • The scroll bar now indicates the location of errors/warnings, search hits, and bookmarks.
  • The xterm title is update to reflect the file name for the top line in the view.
  • Added a "headless" mode so that you can execute commands and run SQL queries from the command-line without having to do it from the curses UI.
  • When doing a search or SQL query, any text that is currently being displayed can be tab-completed.
  • The '-H' option was added so you can view the internal help text.
  • Added the 'g/G' hotkeys to move to the top/bottom of the file.
  • Added a 'log_mark' column to the log tables that indicates whether or not a log message is bookmarked. The field is writable, so you can bookmark lines using an SQL UPDATE query.
  • Added syntax-highlighting when editing SQL queries or search regexes.
  • Added a "write-json-to" command that writes the result of a SQL query to a JSON-formatted file.
  • The "elapsed time" column now uses red/green coloring to indicate sharp changes in the message rate.
  • Added a "set-min-log-level" command to filter out log messages that are below a given level.

You can check out this release on GitHub to get more information and links to the available downloads.

Changes To The Scrollbar

I've made some changes to the scrollbar that is shown on the right side of the display based on some feedback from users.  The scroll area now has a single vertical line extending from the top to the bottom.  Previously, this area would show log message characters and it wasn't very clear that the scroll bar existed.  The line is colored based whether there are errors or warnings in that part of the log.  The coloring should make it easier to see the distribution of errors across the whole log.  Similarly, there are notches added to the left and right side of the line to show search hits and bookmarks, respectively.  See the following screenshot to get an idea of what it looks like:

Screenshot of the redesigned scrollbar while viewing a syslog file.  The line on the right is the scroll area.  The red coloring shows where there are errors in the log.  The notches pointing to the left show the locations of the four search hits.  The notches on the right show the two bookmarks.

Screenshot of the redesigned scrollbar while viewing a syslog file.  The line on the right is the scroll area.  The red coloring shows where there are errors in the log.  The notches pointing to the left show the locations of the four search hits.  The notches on the right show the two bookmarks.

These changes are currently only in the latest code from git.  I'll be playing with things a bit more before making a release.

Word-wrap support in v0.6.2

As requested by Cristobal Dabed and a few others, word-wrap support has been added to version 0.6.2 of lnav.   You can toggle word-wrapping for the log file and text file views by pressing CTRL-W or by using the enable-word-wrap and disable-word-wrap commands.  If you would like to permanently enable word wrapping, you can use ":session enable-word-wrap" to save the command in the session file that is read on startup.

This release also includes some fixes for issues when building/running on Mac OS X Mavericks.

I also want to say that you shouldn't be shy about filing GitHub issues or posting in the lnav google group.  I won't be able to implement every feature request, but I'll do what I can to get lnav more widely adopted.  Thanks for your support!

Competing with "tail -f"

Probably the toughest competition for lnav is the standard Unix utilities like tail, grep, less, and emacs/vim.  It can be hard trying to convince people that these built-in commands that they've used for forever can be improved upon.  The advanced features of lnav might even work against it since folks are expecting to have to learn a bunch of stuff to see any benefits.

The reality is that there are quite a few "passive" features in lnav that can provide value with no effort required by the user.  For example, lnav can easily replace 'tail -f', it's even shorter to type!  Beyond the basic task of displaying new lines appended to a log file, you also get to see log messages from multiple files interleaved, the ability to scroll backwards, syntax highlighting, live searching, and so on.  These basic features do not have the same "wow" factor as executing a SQL query over data automatically extracted from a log file, but they're the features that get used 90% of the time.

Anyways, I think I'm gaining a new appreciation for marketing/sales... 

Support for JSON-encoded logs in v0.6.1

Making logs easily digestible by machines is becoming a concern as tools like elasticsearch become more popular.  One of the popular strategies is to encode the whole log message in JSON and then write that as a single line to a file.  For example:

{"time": "2013-09-04T23:55:09.274041Z", "level" : "INFO", "body" : "Hello, World!" }
{"time": "2013-09-04T23:56:00.285224Z", "level" : "ERROR", "body" : "Something terrible has happened!", "tb": "  foo.c:12\n  bar.y:33" }

Unfortunately, what is good for a machine is not so great for a human.  To try to improve the situation, the latest release of lnav includes support for parsing JSON log messages and transforming them on-the-fly.  The display format is specified in a log format configuration and can specify which fields should be displayed on the main message line.  Any unused fields that are found in the message will be displayed below the main field so you don't miss anything.

The above log lines can be transformed using the following format configuration:

{
    "json_ex_log" : {
        "title" : "Example JSON Log",
        "description" : "An example log format configuration for JSON logs",
        "json" : true,
        "file-pattern" : "test-log\\.json.*",
        "level-field" : "level",
        "line-format" : [
            { "field" : "time" },
            " ",
            { "field" : "body" }
        ]
    }
}

After copying this config to ~/.lnav/formats/test/format.json, the log messages will look like this when viewed in lnav:

2013-09-04T23:55:09.274041Z Hello, World!
2013-09-04T23:56:00.285224Z Something terrible has happened!
  tb:   foo.c:12
  tb:   bar.y:33

lnav v0.6.0 released!

It's been four months since the last release of lnav  and a lot of changes have been made in the mean time.  The most exciting change, for me, is the addition of a generic log parser that will try to automatically extract data from log messages.  For example, lnav can parse the following sudo syslog message and pull out the key/value pairs for TTY, PWD, USER, and COMMAND:

Jun  3 07:00:23 Example-iMac.local sudo[2326]:    ghost : TTY=ttys002 ; PWD=/ ; USER=root ; COMMAND=/bin/ls

This requires no intervention from the user and does not require lnav to know about these log messages.  Rather, the parser searches for text that looks like "<key>=<value>" or "<key>: <value>" and makes that data available through the SQLite front-end built into lnav.  The parser can even handle some text messages if the data values are easily distinguishable from plain text.  You can read more about the parser at the new online documentation site.

Augmenting the generic log parser is the ability to define your own log formats and the addition of a few more built-in formats.  The log formats are now defined in JSON configuration files so it's much easier to add support.  The configuration files contain the regular expressions used to parse the log messages as well as the metadata that describes the values extracted from the log.  If you develop a format for a common log format please send it in for inclusion in future releases.

The user interface has also been cleaned up and should look a bit nicer.  To make learning the hotkeys a bit easier, the bottom right corner will display hints for the hotkeys to use when navigating around.  And, there's a new web site to round out the aesthetic improvements. 


EDIT: I've updated the linux binary download to v0.6.0a to fix an issue with the executable exiting immediately because the terminfo files could not be found.