Are you logging? Do your servers, firewalls, routers, IPS systems, and applications write log files? Good. What are you logging? What do you do with those log files? Do you review them every day? Do you correlate them? Do you investigate errors or strange access patterns? Do you archive them? Or do you just let them take up disk space until you are running low, and then pull the old CTRL+A, Shift+Del to free up space?
Good event log management is about much more than turning on logging and ticking all the boxes. Those logs actually have to be reviewed, archived, consulted, or there’s just so much additional overhead to your I/O, and the equivalent of binary junk taking up precious space until you come along to clear them out. Don’t be a data hoarder; implement good event log management practices to get the most out of your systems.
Here are nine fundamental tips for event log management to help you get started:
1. Use an application to do the heavy lifting for you
Unless you have a very small number of servers, you’ll find you have too many systems to effectively handle event log management by hand. The most important tip for event log management is to use an event log management application. The automation will make event log management scalable, and it will help with the remaining tips in this article.
2. Log only what you need, which is just enough to reproduce the events
Too much information is worse than not enough. It’s not uncommon to find servers configured to log so much that they cannot store more than a rolling 24 hour period worth of data. If someone wants to know on Monday morning what happened Friday night, that data has already been lost. Good event log management avoids information overload by ensuring only the relevant data is logged.
3. Aggregate, and correlate your logs
That event log management software will save you countless hours of logging on to each individual system and trying to gather all the logs manually, and then massaging them in Excel to correlate events. You want to see what happens and when it happens across all your systems, and correlating events is the way to get the big picture.
4. Review the logs regularly
Reviewing logs when you have a problem is a failing strategy. Regularly reviewing logs lets you start to recognize what is normal, so you will notice what is bad. You need to establish that baseline. Regular reviews can also help you spot issues before they become incidents, and that is one of the main reasons to do any kind of event log management at all. Otherwise, you might as well just turn off logging completely to save space.
5. Investigate anomalies
Because you are doing regular reviews as part of your event log management, you will be able to spot anomalies and get ahead of any potential issues before they become major incidents. Whether it is response times, capacity challenges, or inappropriate access attempts, early detection is key.
6. Make the logs accessible, and lead people to them
Don’t go at it alone. Good event log management is a team effort, and you’ll be pleasantly surprised at how well the other server admins and application developers will contribute to regular reviews, because they will start to learn things about their systems they couldn’t possibly know without seeing those logs from production.
7. Mine the logs for useful data
Does it make sense to develop content that will never be used? Of course not, and mining your logs for useful data can help you determine what clients are visiting your website so you can see which browsers and resolutions they are using. It can tell you which sections of a site or application are being used, and which are being ignored. Knowing what is useful to your audience helps you prioritize your development efforts where they will do the most good.
8. Store logs long enough to be able to go back in time to prove what you need
Unless you are trying to show year over year trends, data more than 90 days old is so stale it starts to smell bad. Your mileage may vary, but if a security log shows an access attempt from last month, but the source IP address cannot be correlated to the client because DHCP logs only go back seven days, you have a problem. Good event log management should have prevented that, but can fix the issue going forward. Pick a date that everyone agrees is reasonable, and then make sure that all systems use the same retention period.
9. Purge old logs regularly
And when that retention period is past, purge old logs automatically. Log files take up space, and good event log management includes knowing when to take out the trash, and making sure it gets done.
With these nine fundamental tips for event log management, you will find that logging becomes a valuable and desirable activity that can help you learn about your systems, spot trends that will allow you to plan for future growth, identify issues before they become incidents, and enable you to set management’s expectations properly, and meet them completely. Event log management is an underappreciated skill that can easily be added to your repertoire with the right approach, and a good event log management application to help you with the tasks.