Performing Complex Searches and Reporting on Application Log Files

Abstract

System administration duties often include installing and supporting various applications. These may be either produced by open-source communities or developed in-house. There are also a wide variety of languages used when developing those applications; common languages found these days would be Java, PHP, Python, Ruby and (yes, some still are using it) Perl. In this chapter I am going to talk about applications developed in Java as this seems to be the most common language selected by large enterprises for their web applications. Java applications commonly run within the application server container, such as Tomcat, Jetty, Websphere, or JBoss. You as a system administrator need to know whether the application is running correctly. Every well-organized and structured application is supposed to write its status to one or more log files; in the Java world, this is usually done via the log4j adapter. By observing the log file, a system administrator can detect any faults and failures within the application, which are commonly logged as exception stack traces. The logging of a full exception stack trace usually indicates an unrecoverable error—an error that the application was not able to handle itself. If you do not happen to have many requests, and the application is merely doing anything, catching these exceptions and analyzing them can be done by hand. However, if you need to manage hundreds of servers and there are tens of GBs of information produced, you surely need some automated tools to gather and analyze the data for you. In this chapter I am going to explain how I developed the open-source tool called Exctractor (no, this is not a typo, the name is constructed by joining two words: exception and extractor) and how it functions.