When starting the career of application developer in IT with one or more programming languages, one thing which is kept on back burner and kept for really long is logging. In fact, when I started my career as a programmer, when I realized that I wanted to know about what to log and what not to log, it was almost 4 years or so. This was my involvement in learning nitty-gritties of programming language itself, and not paying enough attention to logging at all. And, when I learnt most of it all, I realized how much value I added to programming by logging appropriate stuff in log files which was used for various different purpose during the execution of the software. One other challenge that comes with logging is what format to use for logging different kind of messages?
Before we go into what to log, lets see some general logging best practices in terms of usage of tools:
- Use frameworks such as Log4J or LogBack (for Java) rather than printing to console. With frameworks, you get some inbuilt advantages such as time, and context information logged out of box without you required to take care of it. With usage of frameworks, you could turn on and off the logging appropriately. This feature comes very handy.
- For those, who are programming with Java, SLF4J serves as a simple facade or abstraction for various logging frameworks (e.g. java.util.logging, logback, log4j) allowing the end user to plug in the desired logging framework at deployment time. Remember that SLF4J is just a facade and not a logging framework in itself. As per SLF4J FAQ page, libraries and other embedded components should consider SLF4J for their logging needs because libraries cannot afford to impose their choice of logging framework on the end-user. Thanks to Francois who pointed out in his comment.
- Try and agree upon a structure or format of logging messages. One of the common format suggested is to log the data in key=value pairs where you could agree on different classes of keys. The keys can be class, method, parameters etc. For example, for method entry, you could log something like method=createCard, parameters={userId=200, orgId=ABCHS, amount=10.0,fees=0.0}. This removes confusion between developers for what to log and also reduces grammatical mistakes owing to lack of English skills of the developers. This becomes much more helpful if your organization or your client is using tools such as SPLUNK for analyzing logs. They get a pattern right away which is easy to search.
- Avoid excessive logging. Logging on each and every step is of very less use. Instead, it tends to create IO bottlenecks.
- From application security perspective, one may want to log “who (userId, transactionId), when (event date & time), what (type, severity), where (source such as application/service/method name)”.
- Plan to categorize logs as information, warning, or error while logging the messages. Based on logging frameworks that you use, this helps you to filter logging and debug faster & better.
- Use class level loggers instead of just one global logger across all the classes.
- Some of the good concepts to learn in relation with logging are filtering, formatting, thread context logger variables. rolling logs, multiple logger targets etc. Thanks to one of our other reader “infernoz” for his comments.
Lets try and see what are some of the minimum events which one would want to log:
- Method entry with input parameters & exit
- Exceptions: Not all exceptions shall be logged, or, in other words, if you want to log exceptions, log it with appropriate trace level such that as and when required they could be written in the logs. Good thing can be to have a custom exception handler component which does the task of logging the exceptions rather than application developer to use their own discretion to do the logging for exceptions. This custom exception handler can have policies around which exceptions to log and which to ignore.
- Query (SQL) Execution: All queries execution can be logged.
- HTTP Requests & Response: Http requests and response can be logged to keep a track on what came as a user request and what was sent as response. This helps a lot in forensics.
- From security perspective, consider logging following:
- Authentication and authorization success and failures
- User sessions start and end.
I have been recently working in the area of Data analytics including Data Science and Machine Learning / Deep Learning. I am also passionate about different technologies including programming languages such as Java/JEE, Javascript, Python, R, Julia, etc, and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data, etc. For latest updates and blogs, follow us on Twitter. I would love to connect with you on Linkedin.
Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking. Check out my other blog, Revive-n-Thrive.com
Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking. Check out my other blog, Revive-n-Thrive.com
Latest posts by Ajitesh Kumar (see all)
- Model Parallelism vs Data Parallelism: Examples - April 11, 2024
- Model Complexity & Overfitting in Machine Learning: How to Reduce - April 10, 2024
- 6 Game-Changing Features of ChatGPT’s Latest Upgrade - April 9, 2024
2 remarks:
* first, talking about Java logging without mentionning SLF4J is an error. Without any specific requirements, you should use SLF4J by default. I mean, you want to be able to see the logs provided by the 3rd party libraries you will be using don’t you ? You want to be able to switch easily from one implementation to another if needed ? So let’s use SLF4J in the code and use any implementation that suits you (Logback, Log4J…)
* second point is the assertion that “Not all exceptions shall be logged”. As the name states, “Exceptions” are meant to be… well… exceptional ! And by definition, when womething “exceptionnal” happens in your programm, you may want to have a way to know it. So my advice would be to log all exceptions by default. Maybe it would be better to just use a lower log level such as trace. Beleive me, when you’re looking for the cause of a bug, you will curse the programmer that omit to log an exception that happen to be exactly what you were looking for.
Thanks for your comments, Francois.
+1 for SLF4J, and Logback blows Log4J away for speed, ease of use, configurations, and features; I’m migrating off Log4J as fast as I can.
Use logging levels and have a firm policy on what level of detail is present at each level, and do have useful debug and even trace logs, so that you can enable lots more detail live, when a knotty issue crops up.
Use at least class level loggers, never just one logger! Also allow external configuration of individual loggers, because you may want to see more detail in specific areas.
Learn about the logger configuration and important things like filtering, formatting, thread context logger variables. rolling logs, multiple logger targets etc.
Very useful comments. I personally have worked with both Log4J and Slf4J and found myself & my teams moving to Slf4J for various different advantages it brings on the table.