Create array from log files and perform queries 1

[czarj]

All-

I'm trying to parse through a bunch of log files and perform some calculations. What I need to do is pipe this data into an array (one line at a time) and then query the array. For example I would be performing operations such as:

i. writing out each section (ndmpd:77, ndmpd:28, ndmpd:99) to an indivudual file or to a single file with blank lines between seach section
ii. searching for ceratin error messages and writing the sections (ndmpd:77, ect) that contain errors to a given file
iii. querying the number of times a given error occurs
iv. ...many more...

If I open the file with a WHILE statement, how then to I get it into an usable array and pull data out of it?

use Getopt::Std;
getopts("i:o:");
if (($opt_i) && ($opt_o)) {} else {print " usage: -i <ndmpdlog> -o <output dir>\n"; exit}

#### open the NDMP log
open (NDMPLOG,"< $opt_i") || die " can't open file $opt_i";

#### check the output dfirectory doesn't already exist
if (-d $opt_o) {
        print " the output directory $opt_o must not exist!\n";
        exit;
}
else { mkdir $opt_o }
mkdir "$opt_o";

#### create and open the ouput file
open (PARSEFILE, ">$opt_o/ndmp.out");

#### parse the ndmp log and output the results
while (<NDMPLOG>) {
 ..............where to go from here..............

---------------------------------------------------------------------------------------
----------------------------------Log File Data----------------------------------
---------------------------------------------------------------------------------------
Feb 26 00:00:28 EST [ndmpd:77]: ReplySequence 7
Feb 26 00:00:28 EST [ndmpd:77]: Error NDMP_NO_ERR
Feb 26 00:00:28 EST [ndmpd:77]: Error sending notify shutdown message
Feb 26 00:00:28 EST [ndmpd:77]: Ndmpd session closed successfully
Feb 26 00:00:28 EST [ndmpd:77]: Error code: NDMP_NO_ERR
Feb 26 00:00:41 EST [ndmpd:84]: NDMP message type: NDMP_DATA_GET_STATE_V2
Feb 26 00:00:41 EST [ndmpd:84]: NDMP message replysequence: 28
Feb 26 00:00:41 EST [ndmpd:84]: Message Header:
Feb 26 05:07:18 EST [ndmpd:63]: Reason: 1
Feb 26 05:07:18 EST [ndmpd:63]: version: 4
Feb 26 05:07:18 EST [ndmpd:63]: Text: Connection shutdown
Feb 26 05:07:18 EST [ndmpd:63]: Ndmpd session closed successfully
Feb 26 05:07:19 EST [ndmpd:63]: Error code: NDMP_NO_ERR
Feb 26 05:07:20 EST [ndmpd:64]: Created an NDMP server connection
Feb 26 05:07:20 EST [ndmpd:64]: Message NDMP_NOTIFY_CONNECTION_STATUS sent
Feb 26 05:07:20 EST [ndmpd:64]: Message Header:
Feb 26 05:07:20 EST [ndmpd:64]: Sequence 1
Feb 26 05:07:28 EST [ndmpd:108]: NDMP message type: NDMP_DATA_GET_STATE_V2
Feb 26 05:07:28 EST [ndmpd:108]: NDMP message replysequence: 498
Feb 26 05:07:28 EST [ndmpd:108]: Message Header:
Feb 26 05:07:28 EST [ndmpd:108]: Sequence 0

--- You must not fight too often with one enemy, or you will teach him all your tricks of war.

 

[prex1]

After your ending [tt]while(<NDMPLOG>){[/tt] you get one line at a time, that you can process as you like, however from your question it is unclear what you want to do.
I suppose the first thing to do is to identify the section, that you can do with (assuming no other square brackets can occur in the data):

  chomp;
  my($date,$section,$content)=split/[\[\]]/;
    #OR if you want the section only
  /\[(.*)\]/;
  my$section=$1;

From then on it's up to you...

Franco

http://www.xcalcs.com

: Online engineering calculations

http://www.megamag.it

: Magnetic brakes for fun rides

http://www.levitans.com

: Air bearing pads

 

[czarj]

Thanks, this seems to work... One question though, what is the best way to remove the extra colon after the bracket?

This code:

while (<NDMPLOG>) {
   chomp;
   my($date,$section,$content)=split/[\[\]]/;

print PARSEFILE "$content\n";

Results in the content section having a leading colon:
--------------------------------
: ReplySequence 2
: Error NDMP_NO_ERR
: Error code: NDMP_NO_ERR
: Vendor:
: Product:
: Revision:
--------------------------------

Thanks Very Much!
Justin


--- You must not fight too often with one enemy, or you will teach him all your tricks of war.

 

[prex1]

If the colon (and the subsequent space) is always there, the fastest is to do
[tt]$content=substr($content,2);[/tt]
Another method:

  my($date,$section,$content);
  ($date,$section)=split/\[/,$_,2;
  ($section,$content)=split/\]:\s*/,$section,2;

This one works even if there is occasionally no space (or multiple spaces) after the colon.

Franco

http://www.xcalcs.com

: Online engineering calculations

http://www.megamag.it

: Magnetic brakes for fun rides

http://www.levitans.com

: Air bearing pads