Most Popular Reports for log analysis tools

Summary of the " Most popular reports...?" thread initiated by Marcus J. Ranum on the Log Analysis mailing list:

"I'm trying to build a list of the "most popular reports" that people pull from their system logs. This is mostly for my curiosity,but also to see if log analysts tend to share common goals, or whether we're all over the spectrum. I'm also hoping to be able to maybe assemble a "top ten" list that people can look/ask for from log analysis vendors.
(I am teaching a tutorial on system log analysis for SANS and USENIX/LISA and will gleefully add your good suggestions to my list!)"

Author	Comments
mjr	Here's my list: N should be considered a settable parameter - Top N machines sending/receiving traffic through the firewall - Top N machines sending/receiving traffic on the network segment same as above but inward-looking - Top N machines being accessed behind the firewall - Breakdown of traffic through firewall by service (percentage) this popular as a pie chart - Breakdown of traffic on the network segment by service (percentage) same as above but inward-looking - Top N email address(es) sending Email messages - Top N email address(es) receiving Email messages - Percentage of Email that is identified as spam - Percentage of Email that contains blocked attachments - Top N machines accessing web - Top N targets identified in IDS alerts - Top N IDS attacks identified - Percentage of web traffic aimed at sites on porn blacklist - Percentage of traffic aimed at sites on spy/adware blacklist - Top N porn-surfers - Top N most-ad/spyware infected systems - New machines that have served WWW/FTP/SMTP today
tbird	- All lines of log data longer than N characters (<if it's not a web server with lots o' really long URLs>) - Summary of remote access usage -- SSH, VPN, telnet (ick) whatever, with at least username, source, destination (to look for people in odd places and check for trends) - ideally, I'd get a "someone logged in from somewhere new" summary, but I'll settle for everything, at least at first - Lines containing "root" and "passwd" and "null" (or perhaps, combinations thereof) - Top N most common lines
Devdas Bhagat	- Top N blocked spam sending systems (by IP, by domain and by ip whois)- Useful for judging how good/bad a DNSBL is with respect to your requirements - Top N servers handling the mail load - Ideally, this should be equally well distributed. - Top N ports being probed - New attacks? - Top N hosts trying to relay via your servers. - Zombies? Compromised hosts? - Top N changes to routing information, if running a dynamic routing protocol. -- Shows network stability/instability - Top N machines spewing out useless traffic on the network. -- essentially a breakdown by protocol per host, rather than by host per protocol. This should indicate protocols which you would try to avoid on the network/other problems. E.g., if a host which normally does 1 Mb/s of NetBIOS traffic suddenly jumps to 10 Mb/s, you have a problem. - New programs installed on various systems. - Any users created/deleted. - Database logs (for tuning purposes -- number of times a specific query is run, how long the query takes, top N queries, etc). - DNS - Top N domains queried for. - If running an authoritative reverse DNS, the number of rDNS queries per host. A sudden spike in the number of rDNS requests could indicate a spam run.
Joe Wulf	Another vein of thought to consider is the "insider threat". In this area, I'd be interested in "access failures". Failure to: - Access a server (login) - Access an application or database - Access a service or resource - Access a directory or file And especially how often and from whom/where these occur.
mjr	Re: tbird's comments: "summary of remote access usage -- SSH, VPN, telnet" Ok, that's a hot one. Let's just call that "VPN usage" frequency, user, etc - same as for a firewall, basically, except out the VPN interface.
Bruce B. Platt	Internet: 1. Top N machines generating Postfix rejects due to forged client name in received header, unknown user name in alias tables, etc. (Postfix and allied filtering mechanisms) do lots of processing outside of our firewalls. 2. Top N domains/IPs generating spam and offering virus-laden mail 3. Snort reports for top N machines trying things like MS IIS exploits 4. Top N sources of probes for ports like ssh, telnet, 135-139 and 445, (lots of overlap between these and #2. :-) and other interesting ports (pick your list) (These are all blocked.) 5. Top N machines trying a dns zone x-fer other than my allowed secondaries. 6. Top N machines looking for smtp servers other than those advertised in domain MX records. (Not really sure why I like this, but it seems smart to me.) Inside: 1. Top N web-browse targets. 2. Top N machines trying to mail to internet (blocked by policy, but a good indicator of compromise). 3. Top N machines trying bot channel ports (blocked by policy, but ditto). 4. Top N machines trying to do ftp-puts (blocked by policy, and no one should be, but ...). 5. Top N users of web proxy (more to see who is doing lots of web browsing than for any specific security reason. 6. Top N dhcp lease requests. (dhcp server supposed to only give out by MAC address, but never hurts to check.) 7. Top N machines needing patches
Jose Nazario	- Trend analysis - can yield more insightful results, but you have to have a decent window and additional filters in place to spot real trends (as opposed to the normal ebb and flow of traffic).
John Kristoff	- Top DNS RR queries. In addition to see what your most popular queries are, if logging recursive queries, you often find 'bot' when you see suspicious looking host names trickle up to the top. This is often because the name has been closed and client resolvers are too dumb not to keep asking for it continuously. - Log message count per hour (or whatever time interval) - for many systems logs across a 24-hour period are very smooth, spikes in any interval period indicates an anomaly. - Top 'unknown/uncategorized' messages - in a couple of the top N summarization tools I've written, I also create a 'top N' unknown or uncategorized section. Ideally N should be greater than the total number of unknown/uncategorized types so you can spot strange messages and investigate.
Jason Haar	- Internal IP addresses being blocked via an edge firewall more than N times/hour Always an indicator on bad activity. Obviously in our Windows-world, effectively all machines are blocked at some point, but more than 200/hour should be pretty safe as a cutoff point to get rid of those False Positives and still catch the bad machines.
Anton A. Chuvakin	- Bottom N Accessed Ports - Bottom N Event Types - Bottom N ... 'show me the relevant oddities from my logs.
Toby Kohlengerg	- Anything that changes significantly - Don't tell me when you see some random event, tell me when the number of events of a specific type increases by 50%. That give me 0->1, 1->2, 2->3, 3->5, 100->150, etc... Which means that I catch all the rare events and I catch the large changes in the noisy events.
Phil Hollows	- Statistical exception analysis of unusual lack of activity as well as a spike -- a deviation in "normal" in either direction is noteworthy
Adrian Grigorof	- Firewall management sessions (successes and failures) - Comparison between 2 time intervals (i.e. ability to generate a report comparing "today" with "yesterday" or "June" with "July"). The report should compare all the relevant statistics for the 2 intervals - Port scans from IP addresses in the "neighbourhood" - always an indication of a worm roaming through your ISP network - Internal computers using a large number of protocols (may indicate some internal user performing network scans)