In the previous blog post we looked at the importance of event logs and all the useful information that event logs provide us with.

How does one access these logs?

Windows Events

As stated previously on the windows platform the central repository for the logs is the Event system. Once can access this by accessing the Administrative Tools under the control Panel. What we’re looking for here is the Event Viewer

Event Viewer: 5th Option from the Top

When opening the Event viewer we are presented with a huge list of events. This is in fact the first challenge we will need to face. While there is a lot of extremely useful information hidden inside the logs, there are also a lot of events/logs which are of no interest to us. There are two ways to mitigate this, either tweak the system to generate logging that is of interest to you or else use a solution that will filter out the noise and provide you only with events you are interested in.

Once event viewer is opened one will find a large amount of events split into a number of categories. Browse around, you are sure to find something interesting. This can also be a good place to start to debug those hard to track issues, such as your account mysteriously locking up after changing your password. At first glance everything seems to be going fine, no errors have been popping up and if it weren’t for the fact that your account is locked up you wouldn’t even know something is wrong. Browsing through the events you might come across the event illustrated below. Now we know that a particular application is failing to authenticate with the ISA Server probably using the old credentials. From there it is easy to deduce that this application is running a service using your account to access the internet and its failed retries are causing your account to be locked up. This is one of the many possible events that one might come across when investigating the Event log. In some cases they are also the only option to determine what is causing some unexpected behavior by the machine.

Event illustrating failed login attemtps by an application

Syslog

Syslog is a complex system that allows a lot of flexibility. The syslog echo system consists of three entities:

I.    Syslog Device
II.    Syslog Relay
III.    Syslog Server

The syslog device is basically any system that generates a valid syslog message which it then passes on to a relay or a server.

A syslog relay is a syslog server that instead of storing any received messages relays them on to another Syslog server. This is ideal for either centralizing your logs or for security reasons. Relaying logs instead of storing them locally on the machine generating them helps to protect them in case the machine is compromised.

The syslog server is the end point where the syslog message reaches its destination. Here the message can be stored into a text file, database or even piped into another application.

A default Linux installation generally has a syslog server which stores any log generated on the machine in a text file found at /var/log/messages. A system can also be configured to log different events in different files and logs are generally rotated (archived to a different file and start a new fresh file) however looking into /var/log/ is generally a good bet.

Much like the Windows event viewer opening the file will display a large number of logs generated by different systems.

While the log format can be changed, it generally consists of:
[Date] [time] [host] [application][application pid] [Log Entry]

Browsing through the log one can find a lot of interesting log entries here as well. A log entry such as:
Sep  1 14:15:38 localhost login[5871]: FAILED LOGIN 1 FROM /dev/vc/4 FOR root, Authentication failure

This informs the user that on 1 September at about 2 p.m. someone tried to log on as root (the administrator account) on console and failed his authentication.

Just as with Windows an administrator can be presented with a lot of log entries especially if logs from multiple machines and devices are aggregated in a central location. This problem can easily be mitigated by using applications that can parse and process syslog messages. Many such applications exist from free to commercial.

SNMP

The final major logging mechanism one will come across is SNMP. SNMP is used mostly by devices such as temperature sensors, Firewalls and other such devices. Its echo system resembles slightly that of syslog. You get the device, an agent running on the device and a manager. The device can be anything from a physical device to a software server and does its own operations which generate logs. The agent is sort of a middle man between the manager and the device. It sits on the device and handles communication. The manager is an application that gets the data from the different agents and presents them to the administrator. Unlike Syslog however, SNMP is bidirectional. Primarily it’s the manager that queries the agent for specific information; however, the agent itself can send notifications to the manager called SNMP traps.

That’s where similarities end however. In order to use SNMP one will definitely need a software solution call NMS (Network Management System).

In contrast to the other systems in SNMP you actively monitor specific things as opposed to going through every log entry. Devices that use SNMP would have various properties such as a standalone mail system where one would find properties like:

  • Inbound/Outbound Queue
  • Mail scanning statistics
  • CPU usage
  • Memory usage
  • Mail delivery successes/failures
  • Other such statistics

Additionally on such a device one would expect traps like:

  • Warning if system is overheating
  • Warning when storage space is running out
  • Warning if too many emails are failing
  • Warning if queues are growing too large
  • Other such warnings

The advantage here is that you’re only monitoring specific items and you can have an automatic notification when certain important events occur provided the devices supports traps for it. The disadvantage is that you will need to set up a software solution to handle SNMP and it can get quite complex to configure properly.

Irrespective of what system Servers, desktops and devices you use in your network it is essential that their logging and events are monitored. It may seem like a daunting task at first, one that is better avoid due to its inconvenience to maintain; however, just like switching off pain would be a very dangerous thing to do, so is ignoring logs. Pain might be an inconvenience but it can save one’s life by telling them when something is wrong. In the same manner, logs may be inconvenient to monitor but they can save your network, and save you time and money when an administrator is able to detect an issue in time and act upon it before much damage is done.