×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Construct a hash from a log file.
2

Construct a hash from a log file.

Construct a hash from a log file.

(OP)
A sample log file somewhat look like this:

CODE

x11:x12:x13:...:x1m:value1
x21:x22:x23:...:x2m:...:x2n:value2
x31:x32:x33:...:x3i:value3
x41:x42:x43:...:x4i:...:x4j:value4 

Note that --
1) In each row, the elements are delimited by ':';
2) The number of elements in each row may very, i.e. m, n, i, & j are integers and m != n != i != j.

How to construct a multi-dimension hash whose keys are the elements in rows denoted as xij and values are the last element in each row.

For instance, if a simplified log file was like:

CODE

x11:x12:x13:value1
x21:x22:x23:x24:value2 

then the hash would be like:

CODE

my %cache => (
    'x11'=>{
        'x12'=>{
            'x13'=>value1,
        },
    },
    'x21'=>{
        'x22'=>{
            'x23'=>{
                'x24'=>value2,
                },
            },
        },
    },
); 

I guess the implementation should use recursive call. But I don't know how to implement it.

Thanks for your kind help.

RE: Construct a hash from a log file.

Hi

No need for recursion, a simple loop is enough :

CODE --> Perl

use Data::Dumper;

my %cache = ();

while (chomp(my $line = <DATA>)) {
  my @field = split /:/, $line;

  my $parent = \%cache;
  for (my $i = 0, $l = scalar @field; $i < $l - 2; $i++) {
    $parent->{$field[$i]} = {} unless defined $parent->{$field[$i]};
    $parent = $parent->{$field[$i]};
  }
  $parent->{$field[-2]} = $field[-1];
}

print Dumper \%cache;

__DATA__
x11:x12:x13:value1
x21:x22:x23:x24:value2 

CODE --> output

$VAR1 = {
          'x11' => {
                     'x12' => {
                                'x13' => 'value1'
                              }
                   },
          'x21' => {
                     'x22' => {
                                'x23' => {
                                           'x24' => 'value2'
                                         }
                              }
                   }
        }; 

Feherke.
http://feherke.github.com/

RE: Construct a hash from a log file.

(OP)
Hi Feherke,

Thank you so much for your help!!

But I have some follow-up questions. At first, please take a look at my modified code and input data.

CODE --> perl

#!/usr/bin/perl -w

use strict;
use warnings;
use Data::Dumper;

my %cache = ();

while (chomp(my $line = <DATA>)) {  # This is line 9
  if($line !~ /^#/) {
    my @field = split /:/, $line;

    my $parent = \%cache;
    for (my $i = 0, my $l = scalar @field; $i < $l - 2; $i++) {
      $parent->{$field[$i]} = {} unless defined $parent->{$field[$i]};
      $parent = $parent->{$field[$i]};
    }
    if($field[-2]) { # To filter NULL strings
      # But would it also filter out 0 - zero's? 
      $parent->{$field[-2]} = $field[-1];
    }
  }
}

print Dumper \%cache;

__DATA__
# the 2nd element is NULL
x11::x13:value1
# the 2nd element is zero
x21:0:x23:x24:value2
x31:x32:value3 

The actual output:

CODE --> output

% ./readCache.pl
Use of uninitialized value $line in chomp at ./readCache.pl line 9, <DATA> line 5.
$VAR1 = {
          'x11' => {
                     '' => {
                             'x13' => 'value1'
                           }
                   },
          'x31' => {
                     'x32' => 'value3 '
                   },
          'x21' => {
                     '0' => {
                              'x23' => {
                                         'x24' => 'value2 '
                                       }
                            }
                   }
        }; 

The output I want:

CODE --> output

$VAR1 = {
          'x11' => {
                     'x13' => 'value1'
                   },
          'x31' => {
                     'x32' => 'value3 '
                   },
          'x21' => {
                     '0' => {
                              'x23' => {
                                         'x24' => 'value2 '
                                       }
                            }
                   }
        }; 

The output I expected from my modified codes, but not I wanted:

CODE --> output

$VAR1 = {
          'x11' => {
                     'x13' => 'value1'
                   },
          'x31' => {
                     'x32' => 'value3 '
                   },
          'x21' => {
                     'x23' => {
                                'x24' => 'value2 '
                              }
                   }
        }; 

My questions:
1) Where does that warning come from? How to get rid of it?
2) Why didn't my implementation get rid of --
i) the NULL string, which is what I expected and wanted;
ii) the '0', which is what I expected, but not what I wanted

RE: Construct a hash from a log file.

Hi

Quote (whn)

1) Where does that warning come from? How to get rid of it?
That is because :
  • You turned on warnings. ( By the way, unless you want to turn warnings on/off for certain blocks, the -w switch and use warnings pragma are equivalent. )
  • Personally I prefer to enjoy Perl's flexibility and ignore warnings. Despite that I try to not force the edges too much when giving advices, there are catchy situations I miss.
In other words, at the end of input, the last value returned by the <> operation is undef. Using chomp() on a variable with undef value, gives that warning :

CODE --> command-line

master # perl -e 'chomp $x'

master # perl -we 'chomp $x'
Name "main::x" used only once: possible typo at -e line 1.
Use of uninitialized value $x in scalar chomp at -e line 1.

master # perl -e '$x=undef; chomp $x'

master # perl -we '$x=undef; chomp $x'
Use of uninitialized value $x in scalar chomp at -e line 1. 

To get rid of that warning just move the chomp() call from the condition inside the loop :

CODE --> (fragment)

while (my $line = <DATA>) {  # This is line 9
  chomp $line; 

Quote (whn)

2) Why didn't my implementation get rid of --

i) the NULL string, which is what I expected and wanted;
In that DATA line the field indexes are
0     1    2     3      # from left
x11 :    : x13 : value1
-4    -3   -2    -1     # from right
 

Even more, you tested after the loop, in that moment the key with the empty string was already created.

Better handle it on split() by splitting not on each colon ( : ), but on one or more consecutive colons :

CODE --> fragment

my @field = split /:+/, $line; 

Quote (whn)

ii) the '0', which is what I expected, but not what I wanted
Similarly, in that DATA line '0' was at index -4, not the tested one.

One minor thing. I find that instructions which condition the execution of the loop body are better to do next ( or continue in other languages ) then to control a huge block enclosing the entire loop block.

So finally I this is what I have now :

CODE --> Perl

#!/usr/bin/perl -w

use strict;
use warnings;
use Data::Dumper;

my %cache = ();

while (my $line = <DATA>) {  # This is line 9
  chomp $line;
  next if $line =~ /^#/;

  my @field = split /:+/, $line;

  my $parent = \%cache;
  for (my $i = 0, my $l = scalar @field; $i < $l - 2; $i++) {
    $parent->{$field[$i]} = {} unless defined $parent->{$field[$i]};
    $parent = $parent->{$field[$i]};
  }
  $parent->{$field[-2]} = $field[-1];
}

print Dumper \%cache;

__DATA__
# the 2nd element is NULL
x11::x13:value1
# the 2nd element is zero
x21:0:x23:x24:value2
x31:x32:value3 

Feherke.
http://feherke.github.com/

RE: Construct a hash from a log file.

(OP)
You're the man, Feherke. Many thanks!!

RE: Construct a hash from a log file.

(OP)
At first, my sincere thanks go to Feherke! With his help I have used the implementation suggested by Feherke a few times at work. They are all working very well. But it encountered a problem lately that I cannot understand it at all.

Hence I constructed an example to show the problem that I am having now. Below is my sample code (a bit long, 63 lines):

CODE

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $log = '/tmp/log';
my $dlmt = '_';
my $tail = 't';
my @ver = (3, 3.22, 5.15, 5.16, '5.2.33', 5);
@ver = (3.22, 3.3, '4.4.4.4', '3.5.21', 6, 4); # this is line 11.
# comment out this line in 2nd run
print Dumper(@ver);
&cacheIt(\@ver);
my $info = &readCache();
print Dumper($info);
exit;

sub readCache {
  my %cache = ();
  open(RH, "$log");
  my $lineCnt = 1;
  my $key;
  while(my $line = <RH>) {
    chomp($line);
    if($line =~ /$dlmt/) {
      my @field = split(/$dlmt/, $line);
      my $parent = \%cache;
      my $l = scalar @field;
      for (my $i = 0; $i < $l - 2; $i++) {
        $parent->{$field[$i]} = {} unless exists $parent->{$field[$i]};
        $parent = $parent->{$field[$i]};
      }
      if($#field > 1) {
        $parent->{$field[-2]} = $field[-1]; # This is line 35
      }
      else {
        if(!exists($parent->{$field[-2]})) {
          $parent->{$field[-2]} = $field[-1];
        }
      }
    } # end of if($line =~ /$dlmt/)
    else {
      $key = $line;
    }
    $lineCnt++;
  } # end of while(my $line = <RH>)
  return \%cache;
}

sub cacheIt {
  my $a = $_[0];
  my %h;
  my $hRef = \%h;
  open(WH, ">$log");
  for(my $i = 0; $i <= $#$a; $i++) {
    my @tmp = split(/\./, $a->[$i]);
    my $str = join $dlmt, @tmp;
    $str .= $dlmt.$tail;
    print WH $str."\n";
  }
  close(WH);
} 

The output from first run. Line 11 is taking effect.
Everything is expected.

CODE

./test.pl
$VAR1 = '3.22';
$VAR2 = '3.3';
$VAR3 = '4.4.4.4';
$VAR4 = '3.5.21';
$VAR5 = 6;
$VAR6 = 4;
$VAR1 = {
          '6' => 't',
          '4' => {
                   '4' => {
                            '4' => {
                                     '4' => 't'
                                   }
                          }
                 },
          '3' => {
                   '22' => 't',
                   '3' => 't',
                   '5' => {
                            '21' => 't'
                          }
                 }
        }; 

Below is the output from 2nd run. Line 11 is commented out.

CODE

./test.pl
$VAR1 = 3;
$VAR2 = '3.22';
$VAR3 = '5.15';
$VAR4 = '5.16';
$VAR5 = '5.2.33';
$VAR6 = 5;
Can't use string ("t") as a HASH ref while "strict refs" in use at ./test.pl line 35, <RH> line 2. 

I may have more questions to follow. But let me stop here for now.
Thank you so much for your time and help.

RE: Construct a hash from a log file.

It seems that in the 2nd run you are reading a file that was created by cacheIt() in the 1st run. Can you give us the content of that file (at least line 2)?

http://www.xcalcs.com : Online engineering calculations
http://www.megamag.it : Magnetic brakes for fun rides
http://www.levitans.com : Air bearing pads

RE: Construct a hash from a log file.

(OP)
Hi prex,

Thanks for your post. After having read your post, I know what's wrong. But I don't know how to fix it, yet.

Below is my updated codes. The code in blue are new. Please also note that the line numbers changed too.

CODE

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $log = '/tmp/log';
my $dlmt = '_';
my $tail = 't';
my @ver = (3, 3.22, '5.2.33', 5, 6); # The 1st element '3' breaks the code
#@ver = (3.22, '5.2.33', 5, 6); # this is line 11.
# comment out line 11 in 2nd run
print Dumper(@ver);
&cacheIt(\@ver); 
my $logFileContents = `cat $log`;
print "Log File Contents:\n$logFileContents\n\n";
my $info = &readCache();
print Dumper($info);
exit;

sub readCache {
  my %cache = ();
  open(RH, "$log");
  my $lineCnt = 1;
  my $key;
  while(my $line = <RH>) {
    chomp($line);
    if($line =~ /$dlmt/) {
      my @field = split(/$dlmt/, $line);
      my $parent = \%cache;
      my $l = scalar @field;
      for (my $i = 0; $i < $l - 2; $i++) {
        $parent->{$field[$i]} = {} unless exists $parent->{$field[$i]};
        $parent = $parent->{$field[$i]};
      }
      if($#field > 1) {
        $parent->{$field[-2]} = $field[-1]; # This is line 37
      }
      else {
        if(!exists($parent->{$field[-2]})) {
          $parent->{$field[-2]} = $field[-1];
        }
      }
    } # end of if($line =~ /$dlmt/)
    else {
      $key = $line;
    }
    $lineCnt++;
  } # end of while(my $line = <RH>)
  return \%cache;
}

sub cacheIt {
  my $a = $_[0];
  my %h;
  my $hRef = \%h;
  open(WH, ">$log");
  for(my $i = 0; $i <= $#$a; $i++) {
    my @tmp = split(/\./, $a->[$i]);
    my $str = join $dlmt, @tmp;
    $str .= $dlmt.$tail;
    print WH $str."\n";
  }
  close(WH);
} 

First run. Line 11 takes effect:

CODE

% ./test.pl
$VAR1 = '3.22';
$VAR2 = '5.2.33';
$VAR3 = 5;
$VAR4 = 6;
Log File Contents:
3_22_t
5_2_33_t
5_t
6_t


$VAR1 = {
          '6' => 't',
          '3' => {
                   '22' => 't'
                 },
          '5' => {
                   '2' => {
                            '33' => 't'
                          }
                 }
        }; 

Second run. Line 11 is commented out:

CODE

./test.pl
$VAR1 = 3;
$VAR2 = '3.22';
$VAR3 = '5.2.33';
$VAR4 = 5;
$VAR5 = 6;
Log File Contents:
3_t
3_22_t
5_2_33_t
5_t
6_t


Can't use string ("t") as a HASH ref while "strict refs" in use at ./test.pl line 37, <RH> line 2. 

RE: Construct a hash from a log file.

(OP)
Ok, I fixed the problem - it's somewhat like a hack.

The code is simplified and the blue part is new and looks like a hack to me. I'd appreciate that if someone could polish it.

Thanks!

CODE

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $tail = 't';
my @ver = (3, 3.22, '5.2.33', 5, 6); # this is line 8.
#@ver = (3.22, '5.2.33', 5, 6); # this is line 9.
# Either line 8 or line 9 would work!!
my $info = &a2h(\@ver);
print Dumper($info);
exit;

sub a2h {
  my $a = $_[0];
  my %h = ();
  my @buf;
  foreach my $x (@{$a}) {
    my @field = split(/\./, $x);
    my $last = $#field + 1;
    $field[$last] = $tail;
    my $hRef = \%h;
    my $l = scalar @field;
    for (my $i = 0; $i < $l - 2; $i++) {
      $hRef->{$field[$i]} = {} unless exists $hRef->{$field[$i]};
      $hRef = $hRef->{$field[$i]};
    }
    if($#field > 1) {
      $hRef->{$field[-2]} = $field[-1];
    }
    else {
      push @buf, $x;
    }
  } # end of while(my $line = <RH>)
  if(@buf) {
    foreach my $x (@buf) {
      my @field = split(/\./, $x);
      my $last = $#field + 1;
      $field[$last] = $tail;
      my $hRef = \%h;
      if(!exists($hRef->{$field[0]})) {
        $hRef->{$field[0]} = $field[1];
      }
    } # end of foreach my $x (@buf)
  } # end of if(@buf)
  return \%h;
} 

And the output:

CODE

% ./test.pl
$VAR1 = {
          '6' => 't',
          '3' => {
                   '22' => 't'
                 },
          '5' => {
                   '2' => {
                            '33' => 't'
                          }
                 }
        }; 

RE: Construct a hash from a log file.

In your last but one post, with line 10 in effect, in line 1 of log file you create $cache{3}='t', then in line 2, with
$parent = $parent->{$field[$i]}
where $i=0 and $field[$i]=3,
you set $parent='t' and this is what perl doesn't like, when $parent is then used as a reference.
Having not read the whole thread, it is unclear to me what is the logic of what you are trying to do; also your code is difficult to read with how you use references. You should restart from the scratch with a more readable code.

http://www.xcalcs.com : Online engineering calculations
http://www.megamag.it : Magnetic brakes for fun rides
http://www.levitans.com : Air bearing pads

RE: Construct a hash from a log file.

Hi

Personally I followed the beginning of this thread, but now I am abit lost. So I just rewrote this dumbly. As your crashes on @ver = ('1', '1.2', '1.2.3'); I decided to not reproduce it exactly.

CODE --> Perl

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $tail = 't';
my @ver = (3, 3.22, '5.2.33', 5, 6);
my $info = &a2h(\@ver);
print Dumper($info);

sub a2h {
  my $a = $_[0];
  my %h = ();
  my @buf;
  foreach my $x (@{$a}) {
    my @field = split(/\./, $x);
    my $hRef = \%h;
    my $hPrev;
    my $l = scalar @field;
    for (my $i = 0; $i < $l; $i++) {
      $hPrev->{$field[$i - 1]} = $hRef = {} unless ref $hRef eq 'HASH';
      $hRef->{$field[$i]} = $i < $l - 1 ? {} : $tail unless exists $hRef->{$field[$i]};
      $hPrev = $hRef;
      $hRef = $hRef->{$field[$i]};
    }
  }
  return \%h;
} 
I know, not much readable. So no hard feelings if you prefer to not use it.

Feherke.
feherke.github.io

RE: Construct a hash from a log file.

(OP)
To Feherke,

Actually, I like it a lot and have learned a lot from your posts, including this one!

Thank you very much. Best regards to you.

To prex,

I have already realized what was wrong after I read your first response. Thank you.

>it is unclear to me what is the logic of what you are trying to do
I know and I am sorry. In my first post in this thread, I said what I was trying to accomplish. With Feherke's help, I have achieved my goal.

Then I started to work on a new problem and I thought I cold leverage what I learned from Feherke's post in this thread.

Then I got this error: Can't use string ("t") as a HASH ref while "strict refs" in use at ./test.pl line 37, <RH> line 2.

So I decided to ask for help again in this thread w/o clearly explaining what my goal is. Sorry about this.

With Feherke's help, I have made a little progress. But I have further questions which I'll ask for your help in a new thread today or tomorrow (Time Zone: US EDT).

Sorry for the confusion and thank you so much for your time and help.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close