×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Extract out of Log file

Extract out of Log file

Extract out of Log file

(OP)
Hello

I have a log file which I'm looking to extract certain lines containing keywords so far I have managed to produce the script below which extracts any line containg words EventEstablished and Primay into a text file output as below

11/06/12 09:54:04.549 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:08.752 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:11.330 Trc 04542 EventEstablished sent to [3280] (00002a29 Primary Telephony Server 170.20.10.11:4193)
11/06/12 09:54:11.330 Trc 04542 EventEstablished sent to [3280] (00002a29 Primary Telephony Server 170.20.10.11:4193)
11/06/12 09:54:12.705 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:12.986 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:13.033 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:15.158 Trc 04542 EventEstablished sent to [3280] (00002a29 Primary Telephony Server 170.20.10.11:4193)

The script I use is below:

# Usage: perl Find.pl infile outfile
use strict;
use warnings;

my ($qfn_in, $qfn_out) = @ARGV;

open(my $fh_in, '<', $qfn_in)
or die("Unable to read file \"$qfn_in\": $!\n");
open(my $fh_out, '>', $qfn_out)
or die("Unable to create file \"$qfn_out\": $!\n");

while (<$fh_in>) {

if (/EventEstablished/ and /Portrait/) {
print $fh_out $_;
}
}



Now this is a very basic script I've been trying to enhance it but can't work out quite how yet but I'd like to add the following


1) Ability to add additional lines to extract and include in the output file when additional keywords are availiable, e.g. to also extract lines containing words RequestAgentLogin received

2) Add a header to the output file


3) Display Output file to include only cetain values out of the extracted line and strip down only IP address

i.e. what I would like the output file to produce is something like

Date Time Event IP Address
11/06/12 09:54:04.549 EventEstablished 170.20.10.11
11/06/12 09:54:08.752 EventEstablished 170.20.10.11
11/06/12 09:54:11.330 EventEstablished 170.20.10.41
11/06/12 09:58:11.330 RequestAgentLogin 170.20.10.28



Any help would be greatly appreciated



Thanks

M

RE: Extract out of Log file

Hi

Personally I would prefer to write it shorter as :

CODE --> command-line

perl -ne 'print if/EventEstablished/&&/Primary/' infile outfile 

Which enhanced with the requested functionalities would look like this :

CODE --> command-line

perl -nle 'BEGIN{@l=qw{EventEstablished RequestAgentLogin};print"Date Time Event IP Address"}print"$1 $2 $3"if/Primary/&&/^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/&&$2~~@l' infile outfile 

Feherke.
http://feherke.github.com/

RE: Extract out of Log file

(OP)
Hi Feherke,

Thanks for you reply, I'm just getting to grips with perl so my code is more than likely far from perfect ;)

Ideally I'd like to put everything within the perl file so all i pass through the command line is the name of my perlscript and the input and output files.

I've attempted to run the code you've kindly provided but encountered some issues

Get an error message "Cant Find String terminator "" anywhere before EOF at -e line 1"

Is there anyway to incorporate your enhanced code within a perl script and run the script?

Your code: perl -nle 'BEGIN{@l=qw{EventEstablished RequestAgentLogin};print"Date Time Event IP Address"}print"$1 $2 $3"if/Primary/&&/^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/&&$2~~@l' infile outfile

Does this extract all lines containing words EventEstablised or RequestAgentLogin?
If I wanted to include all lines EventEstablished and Primary or RequestAgentLogin how can this be done via the script?

Thanks again :)


RE: Extract out of Log file

Hi

Quote (mmiah1982)

Get an error message "Cant Find String terminator "" anywhere before EOF at -e line 1"
Interesting. Please specify your operating system, shell/command interpreter and Perl version.

Quote (mmiah1982)

Does this extract all lines containing words EventEstablised or RequestAgentLogin?
Actually extracts this lines which contain "Primary" anywhere and contain something looking as IP address as their 7th or later word and the 5th word is either "EventEstablised" or "RequestAgentLogin".

Feherke.
http://feherke.github.com/

RE: Extract out of Log file

(OP)
Hi

Interesting. Please specify your operating system, shell/command interpreter and Perl version.
-- Windows XP, Perl v5.10.1[/b]

Actually extracts this lines which contain "Primary" anywhere and contain something looking as IP address as their 7th or later word and the 5th word is either "EventEstablised" or "RequestAgentLogin".
----Looking to extract lines containing words EventEstablished and Primary and also lines contaning RequestAgentLogin
(there are other lines within log file just containing word Primary which i dont need, only when it also has EventEstablished aswell i'm interested in)

Thanks Again

M

RE: Extract out of Log file

Hi

Quote (mmiah1982)

Windows XP, Perl v5.10.1
Well, that changes everything.

Quote (mmiah1982)

Looking to extract lines containing words EventEstablished and Primary and also lines contaning RequestAgentLogin
You mean the presence of "Primary" is required only for lines containing "EventEstablished" but not necessary for lines containing "RequestAgentLogin" ?

A complex sample input containing both needed and unneeded cases would really help.

Then I would use this code :

CODE --> Find.pl

use strict;
use warnings;

print "Date Time Event IP Address\n";

while (<>) {
  print "$1 $2 $3\n" if ((/EventEstablished/ && /Primary/) || /RequestAgentLogin/) && /^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/
} 
Like this :

CODE --> command-line

perl Find.pl infile > outfile 

Feherke.
http://feherke.github.com/

RE: Extract out of Log file

(OP)
Thanks Feherke!

I think that's almost exactly what I'm after!!! One additional tweak is at the moment the input file is the name of a single file that you choose, if the input was a list of files in a directory how would i read this in?

So the script would be in the directory containing all the log files which contain the file names in the same format

i.e

Tlog_06102012_1540.log
Tlog_06102012_1440.log
Tlog_06102012_1340.log
Tlog_06102012_1240.log
Tlog_06102012_1130.log
Tlog_05102012_0940.log
etc........

So I need my command to be perl Find.pl Tlog*.lo > OutputFile


So I can better understand your code, do you mind adding in some commentry on your regular expression so I can work out how you selected which columns to show on the output, and also how you stripped off just the IP address?


print "$1 $2 $3\n" if ((/EventEstablished/ && /Primary/) || /RequestAgentLogin/) && /^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/



Thanks Again Really Appreciate your help!
M

RE: Extract out of Log file

Hi

Quote (mmiah1982)

So I need my command to be perl Find.pl Tlog*.log > OutputFile
Well, on Unix-like operating systems that would be the exact command to run.

To make the same work on Windows too, you need to add the globbing to the script :

CODE --> Find.pl

use strict;
use warnings;

print "date time event ip address\n";

@argv = <@argv>;

while (<>) {
  print "$1 $2 $3\n" if ((/eventestablished/ && /primary/) || /requestagentlogin/) && /^(\s+ \s+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/
} 
Warning The above code was not tested on Windows. It works on Linux and the documentation mentions no reason why it wound not work on Windows.

 ,--- starts with
 |  ,-- a non-space character
 |  |,-- previous item 1 or more times
 |  ||       ,-- a word character
 |  ||       |            ,-- any character
 |  ||       |            |,-- previous item 0 or more times
 |  ||       |            ||   ,-- a digit character
 |  ||       |            ||   |  ,-- previous item 1 up to 3 times
 |  ||       |            ||   |  |    ,-- non-capturing group
 |  ||       |            ||   |  |    | ,-- a literal . character
 |  ||       |            ||   | _|_   | |         ,-- previous item 3 times
 | /\|      /\            ||  /\/   \ /\/\        / \
/^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/
  \_______/          \_/      \______________________/
      $1              $2                 $3
       \______________|__________________/
                captured group
 

Feherke.
http://feherke.github.com/

RE: Extract out of Log file

(OP)
Thanks Again for the explanation :)

Adding Globbing didn't seem to work

get following error Global symbol @argv requires explicit package name at Find.pl

M

RE: Extract out of Log file

Hi

Oops. No idea how I managed to post that mess. ( Was a long day yesterday. )

@ARGV should be written in uppercase :

CODE --> (fragment)

@ARGV = <@ARGV>; 

Feherke.
http://feherke.github.com/

RE: Extract out of Log file

(OP)
Perfect! I can't thank you enough for all your help on this, even my noob questions! :)

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close