Thursday, March 13, 2008

Orb Designs Graffiti, a Daily Blog - March 10, 2008 thru March 16, 2008

Ugh. Eyes hurt, nose burns, throat hurts, feel baaaad.

But I pretty much resolved an issue I had been working on at work - involving how best to analyse the firewall logs. The main issue - size. We are getting 15-20 million hits per year, and that increases every year.

For instance, Jan 2006 we got 700k hits. That's a small month. The largest so far is Oct 2006 at about 6 times that.

So I wrote a script to first consolidate the daily log files to one per month after removing the entries we don't care about. Then I read in a file of events we want to count and process all twelve monthly files counting the events in each one and then the total. This creates a nicely formatted text file I can import into Excel and come up with lovely graphs and charts.

Of course, the counting script takes around 2 hrs to run. It was suggested by some friends online that reading the log files line by line and counting the event hits that way would be faster, as there are nearly 100 events being counted, and 12 large files, I was doing 1200 file opens per year, basically.

That went down in flames when after 12 hrs it wasn't finished with the first month's file. heh. Seems that 700,000 lines to read and do an external expr compare is FAR slower than a simple grep -c on a huge file, even done 100 times per file.

I did manage to cut the processing time down from 5 hrs to 2 by changing the procedure from 'cat events.txt | while read event' to 'while read event; do ... done < events.txt'

It seems that using cat is horrendously slow and directly feeding the file to the do-done loop is WAY faster. Good to know.

So now I have the counts and just need to develop some nice charts and graphs.

I did play with fwanalog, and couldn't make it work. Even the new version 0.6.9 that is compatible with our firewall is only compatible with a lower end version of it, not the one we have. I may tinker with writing the required filterset for it later, but for now, what I have will suffice.

I just hope I feel better - I hate being sick. I had just bragged a few weeks ago to some friends that I hadn't been sick all winter because I was taking 1500mg Vitamin C every day. Of course, I forgot to take it for a few days, and then got out of the habit, so after being without it for a week or so, I now feel like a cold is coming on.

I'm up to 2500mg per day now, plus my regular vitamin and B-1000 complex I had also stopped taking. Laziness is my downfall.