What consists of useful logs?

Today I will talk a little bit about logs and what added value these can bring to your day-to-day security operations in addition to just storing them for X amount of months because some regulation says so.

In general logs come from network devices, operating systems and services. Also applications can produce logs but usually these have their own logging. Most can do syslog formatted logging and support logging to a separate log server, and some can utilize an agent which modifies the logs to syslog format. Of course if your environment is homogenous and it supports a specific logging system, use it instead if it meets your requirements. There exists also logging appliances which could be purchased and set up to receive logs from different sources. However, this post is not about the logging infra (which can be easy or hard to deal with), but the content of the logs.

One obvious log to follow is the access log.

Many devices, operating systems and services create access logs, for example a login attempt to FTP service, SSH, Windows login, VPN devices, databases, telnet, router login and so on. Getting in with password guessing is still accomplishable even though the whole password topic has been debated for ages.

A telltale sign that you are in trouble is a lot of failed login attempts against a specific user which finally stops with a successful login. Even though you don't see that successful login, it might be a good time to check the strenght of the passwords in the attacked system.

Also odd login times might mean that someone unauthorized has gained access to an account and is using it for evil purposes. Centralized logs can turn out to be a handy tool when you're doing incident response and are faced with a system which log files are modified or deleted.

Another log source is the firewall log.

Obviously lots of network activity through a firewall will cause loads of log data which can be very cumbersome to analyze. Individual operating systems could also produce packet filtering logs. In smaller environments it should be easier to handle this kind of data, but with some traffic analysis tools larger sets of data can be analyzed. One might think that logging only the denied packets is sufficient, but it can be worthwhile to log the accept packets too, because these can reveal administrative mistakes in the rulebase and other things.

When you are facing a compromized system you might want to know during the investigation if the system has been used to access other hosts in your network. You might find out that some hosts are running unnecessary services or you have made a mistake in the rulebase which allow outbound traffic from your servers which you never intended to. Anyways the logs could be used for correlation in many situations.

How about system/service events?

Logs here could for example reveal that a HDD is breaking down on an important server, as disk i/o errors start appearing in the logs. You could also notice unexpected service restarts or service errors which could indicate a successful attack against your system. Correlating such events with IDS logs and firewall logs helps you validate a possible incident and if you should investigate further. Added users, modifications in groups and so on can also prove helpful, amongst other things like sudden increase of outgoing emails through your email gateway.

Are IDS events any useful?

IDS events trigger when a packet meets a specific signature in the rulebase. For efficiency and removing false positives, a rulebase should be tuned to the environment you're trying to monitor. Sure, it might keep you blind to ongoing (wrong) attacks but eventually trigger to the meaningful ones. The benefit of an IDS is also a lot up to the placement. If you put it in front of a bunch of webservers that like to talk a lot HTTPS, then the added-value is not that good. If you have load-balancers which also offload the SSL from the servers, then you have a better chance of getting something useful if you place one leg behind the balancers.

In addition to the default rulesets you should build custom rules for the assets you're protecting, e.g. webapps. Having IDS log to a central server mainly would log events but what you also need during your incident response is the payload. Of course the IDS event and other related logs for the affected asset give you a quick correlation of the flow of a possible incident, but the payload shows what triggered the signature. Payload is usually viewed through a IDS management console or separate database frontend.

Last thing to talk about is the service application logs.

By the service application logs I mean actual logs produced by a service, like DHCP, WWW or proxy logs. Webserver logs might not be so useful because these usually don't log POST requests and logging those fully could anyways be a privacy issue (e.g. credentials are mostly given over POST instead of GET). They surely could help assessing what a would-be attacker has done on the site before and after the IDS signature was triggered. Proxy logs on the other hand could help you hunt down infected internal hosts which try to talk with outside Command and Control servers over HTTP. DHCP servers would log the hostname of a computer if it has been present in the DHCP request, which can help you track down the actual user of an infected asset.

As you can see from all the examples given above, the focus of logs from my point of view are more for incident response purposes. These can also prove useful for system administration and troubleshooting. I bet you can come up with lots of other intresting log sources (HIDS, AV?), use scenarios and suggestions on where certain type of logs can be beneficial, but this is what came from the top of my head.

Have a nice week!