1. Simple - grep for an error, then pull out the fifth field:

grep "ERROR - Unauthorized Access" error_log | awk '{print $5}' > error_report

2. Slightly more complex - Pull a username out of a gzipped file, then grep again to pull out all ‘PUT’ operations, then print out the third and last field with a space between them, then sort, then uniq -c

zgrep -i compubiz customer_access_log.gz | grep " PUT - " | awk '{print $3 " " $NF}' | sort | uniq -c > compubiz_put_report

3. Fancy example - Think of a log file that looks like this:

20090823 10:45:10 jsmith,192.168.1.54,84.21,www.google.com,Y
20090823 10:53:02 jfrank,192.168.1.21,60.21,www.slashdot.org,Y
20090823 10:54:39 jsmith,192.168.1.54,22.93,www.cnn.com,Y

So, you want to grep for all instances of ‘jsmith’, then grab the comma separated line and pull just the useful stuff out. Since awk uses spaces by default, you have to tell it to split on commas instead. While I could have said ‘print the first and fourth fields’, I chose to say ‘print the first and second-from-last fields’, which comes in handy when lines get really long. Note the second sort, which will do a reverse numeric sort on the numbers printed by uniq -c.

grep jsmith example_log | awk '{print $3}' | awk -F , '{print $1 " " $(NF-1)}' | sort | uniq -c | sort -rn > example_report

Example output:
29181 jsmith www.google.com
  398 jsmith www.cnn.com
   23 jsmith www.foo.com
    1 jsmith www.bar.net

4. Process killer

ps -elf | egrep badprocessname | awk '{print $4}' | sudo xargs kill

5. Quick ‘who’s being naughty?’ Cisco ASA firewall reports

Disgruntled source IPs:
grep "ASA-4-106023" 2009.09.03.local4 | awk '{print $9}' | awk -F : '{print $2}' | awk -F / '{print $1}' | sort | uniq -c | sort -rn

Popular destination IPs:
grep "ASA-4-106023" 2009.09.03.local4 | awk '{print $11}' | awk -F : '{print $2}' | awk -F / '{print $1}' | sort | uniq -c | sort -rn

6. Apache Web Hits

Hits per minute (Apache combined log format)

cat access_log | awk '{print $5}' | cut -c 2- | awk -F \: '{print $1 ":" $2 ":" $3}' | sort | uniq -c

Hits per hour (Apache combined log format)

cat access_log | awk '{print $5}' | cut -c 2- | awk -F \: '{print $1 ":" $2}' | sort | uniq -c

7. Watch log files in realtime for errors

Apache combined log format:
tail -f access_log |  awk '{print $10 " " $1 " " $2 " " $8}' | grep -v ^[2,3]0

Toss out values of one second (or less):
tail -f time_log | awk '{print $6 " " $8}' | egrep -v "^[01] "