Logging Tips/Best Practices for Newbies

When starting the career of application developer in IT with one or more programming languages, one thing which is kept on back burner and kept for really long is logging. In fact, when I started my career as a programmer, when I realized that I wanted to know about what to log and what not to log, it was almost 4 years or so. This was my involvement in learning nitty-gritties of programming language itself, and not paying enough attention to logging at all. And, when I learnt most of it all, I realized how much value I added to programming by logging appropriate stuff in log files which was used for various different purpose during the execution of the software. One other challenge that comes with logging is what format to use for logging different kind of messages?

Before we go into what to log, lets see some general logging best practices in terms of usage of tools:

• Use frameworks such as Log4J or LogBack (for Java) rather than printing to console. With frameworks, you get some inbuilt advantages such as time, and context information logged out of box without you required to take care of it. With usage of frameworks, you could turn on and off the logging appropriately. This feature comes very handy.
• For those, who are programming with Java, SLF4J serves as a simple facade or abstraction for various logging frameworks (e.g. java.util.logging, logback, log4j) allowing the end user to plug in the desired logging framework at deployment time.  Remember that SLF4J is just a facade and not a logging framework in itself. As per SLF4J FAQ page, libraries and other embedded components should consider SLF4J for their logging needs because libraries cannot afford to impose their choice of logging framework on the end-user. Thanks to Francois who pointed out in his comment.
• Try and agree upon a structure or format of logging messages. One of the common format suggested is to log the data in key=value pairs where you could agree on different classes of keys. The keys can be class, method, parameters etc. For example, for method entry, you could log something like method=createCard, parameters={userId=200, orgId=ABCHS, amount=10.0,fees=0.0}. This removes confusion between developers for what to log and also reduces grammatical mistakes owing to lack of English skills of the developers. This becomes much more helpful if your organization or your client is using tools such as SPLUNK for analyzing logs. They get a pattern right away which is easy to search.
• Avoid excessive logging. Logging on each and every step is of very less use. Instead, it tends to create IO bottlenecks.
• From application security perspective, one may want to log “who (userId, transactionId), when (event date & time), what (type, severity), where (source such as application/service/method name)”.
• Plan to categorize logs as information, warning, or error while logging the messages. Based on logging frameworks that you use, this helps you to filter logging and debug faster & better.
• Use class level loggers instead of just one global logger across all the classes.
• Some of the good concepts to learn in relation with logging are filtering, formatting, thread context logger variables. rolling logs, multiple logger targets etc. Thanks to one of our other reader “infernoz” for his comments.

Lets try and see what are some of the minimum events which one would want to log:

1. Method entry with input parameters & exit
2. Exceptions: Not all exceptions shall be logged, or, in other words, if you want to log exceptions, log it with appropriate trace level such that as and when required they could be written in the logs. Good thing can be to have a custom exception handler component which does the task of logging the exceptions rather than application developer to use their own discretion to do the logging for exceptions. This custom exception handler can have policies around which exceptions to log and which to ignore.
3. Query (SQL) Execution: All queries execution can be logged.
4. HTTP Requests & Response: Http requests and response can be logged to keep a track on what came as a user request and what was sent as response. This helps a lot in forensics.
5. From security perspective, consider logging following:
   • Authentication and authorization success and failures
   • User sessions start and end.