Scrub IIS Log? 1

[SMerrill]

Has anyone written an AWK to analyze the IIS log files that look like the example below?
I wish to gather statistics about how often people are logging in to a DEFAULT.ASP, grouped by IP address.
Thanks!

#Software: Microsoft Internet Information Server 4.0
#Version: 1.0
#Date: 2001-07-17 20:53:48
#Fields: time c-ip cs-method cs-uri-stem sc-status
20:53:48 156.74.138.70 GET /Default.asp 200
20:53:48 156.74.138.70 GET /iissamples/default/SQUIGGLE.GIF 200
20:53:48 156.74.138.70 GET /iissamples/default/MSFT.GIF 200

--Shaun Merrill

 

[vgersh99]

something like that:

nawk -f iis.awk iis.log

#------------------ iis.awk
BEGIN {

FLDtime="1"
FLDc_ip="2"
FLDcs_method="3"
FLDcs_uri_stem="4"

Page="/default.asp"
}

/^[^#].*/ {
if ( tolower($FLDcs_uri_stem) == Page )
stat[$FLDc_ip]++;
}

END {
for (i in stat)
printf("%-8d %s\n", stat, i);
}


vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[SMerrill]

Vlad, you're awesome!

Now for a mod: These files are separated into one file per day with the filename EX[red]YYMMDD[/red].LOG.

How do I nest this procedure into an AWK that will loop through every input file and gather that same statistic?

I am in a Windows enviroment.

Thanks,


--Shaun Merrill

 

[vgersh99]

do you need stats per day or total for ALL the days/supplied files?

DO I hear a 'yes'?

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[SMerrill]

Per day, please. But the date is the one in the &quot;#Date&quot; line of the file, not the date within the filename.

I noticed that the following line of code makes me a file with all the filenames in it. I am sure you can somehow magically suck this in and use it to inhale all the files.

dir /B *.log > dir.txt

The results look like this:

ex010810.log
ex010811.log
ex010813.log
ex010814.log
ex010815.log
ex010816.log

 

[vgersh99]

ok, something like that:

nawk -f iis.awk *.log

I'm not sure how Windows doe sthe 'wild-card' expansion, but it shoud get you going - try it.

#------------------ iis.awk

BEGIN {

FLDtime=&quot;1&quot;
FLDc_ip=&quot;2&quot;
FLDcs_method=&quot;3&quot;
FLDcs_uri_stem=&quot;4&quot;

Page=&quot;/default.asp&quot;
}


/^#Date:/ { fileDate=$2 }

/^[^#].*/ {
if ( tolower($FLDcs_uri_stem) == Page )
stat[fileDate,$FLDc_ip]++;
}

END {
for (outterI in stat) {
split(outterI, outterA, SUBSEP);
printf(&quot;fileDate->[%s]\n&quot;, outterA[1])
for (innerI in stat) {
split(innerI, innerA, SUBSEP);
if (outterA[1] == innerA[1]) {
printf(&quot;\t%-8d %s\n&quot;, stat[innerI], innerA[2]);
delete stat[innerI]
}
}
}
}


vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[SMerrill]

YES, This works incredibly well!!

This works if I execute the line

awk -f iis.awk ex030421.log

but I have yet to figure out how to get the contents of every file to stream through.

It is as if I need to take my DIR.LOG file and convert it to

AWK -f IIS.AWK ex010810.log >> OUTPUT.TXT
AWK -f IIS.AWK ex010811.log >> OUTPUT.TXT
AWK -f IIS.AWK ex010813.log >> OUTPUT.TXT

and then execute it as a batch file ... I think I can manage after that.

Thanks again for all your help, Vlad!

--Shaun Merrill

 

[vgersh99]

I just tried [on my test files]:

awk -f iis.awk ex*log

under W2K 'cmd' and it worked just fine.

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[SMerrill]

Just talking to myself. I finished this post . . .

# -------Launch.AWK
// {printf("AWK -f IIS.AWK %s >> OUTPUT.TXT\n",$0)}

then a file called LAUNCH.BAT:

:: LAUNCH.BAT
@ECHO OFF
dir /B *.log > dir.txt
del output.txt
awk -f launch.awk dir.txt > awkit.bat
del dir.txt
call awkit

Then the resulting file OUTPUT.TXT contains:

fileDate->[2003-01-03]
	11       156.74.138.142
fileDate->[2003-01-04]
	5        156.74.138.128
	16       156.74.138.142
fileDate->[2003-01-08]
	8        156.74.138.142
	1        156.74.138.62

Thank you so much for your expertise. You are incredible!

--Shaun Merrill