Creating a hash of buffers

[zipper777]

Is there a way to create anonymous memory buffers and store them in a hash and then write to them and finally flush the buffers to the hard drive? Basically what I'm doing is creating a basic script that reads in a log file and parses it out in to separate files based on an identifier. There will normally be hundreds of identifiers and I found that writing out each line in the foreach(@lines) is very inefficient, since it has to constantly open and close files to write a single line. So my thought was to create a hash of buffers using the identifier as the key and then store a handle to a memory buffer and write to that then flush all the buffers. I wrote the following code to do that, but I seem to me having issues with creating and writing to the buffers. I thought I could handle them just like a file handle, but I'm getting run time errors on trying to write to the buffer:

Can't call method "write" on an undefined value at z:\workspace\pn\pn.pl line 67, <GEN0> line 1

It sounds like the interpreter can't tell what is in the hash value and it needs to be cast to something, but I'm not familiar with Perl to really tell.

sub breakLogThreads(){
	my $filename = $_[0];
	my $inFileHandle = new FileHandle;

	$starttime = time();
	$oldDelim = $/;
    undef $/;
    $inFileHandle->open($filename) or die "Could not open file: " . $filename;
    $var = <$inFileHandle>;
    $/ = $oldDelim;
    my @lines = split /$oldDelim/, $var;
    %bufHash;
    foreach (@lines){
		$line = $_;
		if ($line =~ m#\d{4}\/\d{2}\/\d{2} .*? (.*?) (.*?) .*#){
			if (!exists $bufHash{$2}){
			 	open(MEMORY,'>', $bufHash->{$2});
			}
			$bufHash->{$2}->write($_);
			$bufHash->{$2}->close();
		}
	}
    print "Took: " .  (time() - $starttime) . " Seconds";
    for my $key( keys %hash){
    	open(MYOUTFILE, ">>c:\\logs\\temp\\T" . $key . ".log") or die "I failed";
		print MYOUTFILE $bufHash{$key};
		close(MYOUTFILE);
    }
	$inFileHandle->close();
}

Sorry for the newb question, and thanks for any help.

-Zipper

 

[MillerH]

There's no need to buffer the output yourself, let perl do it for you. The only thing you need to do is make sure that you aren't reopening and closing your file handles all the time. And to do this, just save them in a hash with identifier/handle relationship. Take the following code snippet for example:

[url=http://perldoc.perl.org/functions/use.html][black][b]use[/b][/black][/url] [green]strict[/green][red];[/red]

[url=http://perldoc.perl.org/functions/my.html][black][b]my[/b][/black][/url] [blue]%fh_of[/blue] = [red]([/red][red])[/red][red];[/red]

[olive][b]for[/b][/olive] [black][b]my[/b][/black] [blue]$number[/blue] [red]([/red][fuchsia]0..10000[/fuchsia][red])[/red] [red]{[/red]
	[black][b]my[/b][/black] [blue]$id[/blue] = [blue]$number[/blue] [blue]%[/blue] [fuchsia]10[/fuchsia][red];[/red]
	
	[black][b]my[/b][/black] [blue]$fh[/blue] = [blue]$fh_of[/blue][red]{[/red][blue]$id[/blue][red]}[/red] ||= [url=http://perldoc.perl.org/functions/do.html][black][b]do[/b][/black][/url] [red]{[/red]
		[black][b]my[/b][/black] [blue]$filename[/blue] = [red]"[/red][purple][blue]$id[/blue].txt[/purple][red]"[/red][red];[/red]
		[url=http://perldoc.perl.org/functions/open.html][black][b]open[/b][/black][/url][red]([/red][black][b]my[/b][/black] [blue]$handle[/blue], [red]'[/red][purple]>[/purple][red]'[/red], [blue]$filename[/blue][red])[/red] or [url=http://perldoc.perl.org/functions/die.html][black][b]die[/b][/black][/url] [red]"[/red][purple]Can't open [blue]$filename[/blue]: [blue]$![/blue][/purple][red]"[/red][red];[/red]
		[blue]$handle[/blue][red];[/red]
	[red]}[/red][red];[/red]
	
	[url=http://perldoc.perl.org/functions/print.html][black][b]print[/b][/black][/url] [blue]$fh[/blue] [red]"[/red][purple][blue]$number[/blue][purple][b]\n[/b][/purple][/purple][red]"[/red][red];[/red]
[red]}[/red]

[gray][i]# Close all handles:[/i][/gray]
[olive][b]foreach[/b][/olive] [black][b]my[/b][/black] [blue]$fh[/blue] [red]([/red][url=http://perldoc.perl.org/functions/values.html][black][b]values[/b][/black][/url] [blue]%fh_of[/blue][red])[/red] [red]{[/red]
	[maroon]close[/maroon][red]([/red][blue]$fh[/blue][red])[/red][red];[/red]
[red]}[/red]

[tt]------------------------------------------------------------
Pragmas (perl 5.10.0) used :
[ul]
[li]strict - Perl pragma to restrict unsafe constructs[/li]
[/ul]
[/tt]- Miller

 

[zipper777]

Thanks, that worked perfectly. I did have one question though. What is the ||= operator? I tried to google it, but google won't use that as a valid search parameter, and none of my perl books reference it.

Thanks again for the help.

-Zipper

 

[MillerH]

perlop - Assignment Operators

- Miller

 

[zipper777]

I hate to prove my own ignorance, but I read though that page, but I'm still not sure I understand what it is really doing. The ||= operator looks like it would take the left operand and logical OR it with the right operand and then assign the result back to the left operand. That might be what this is doing, but I'm afraid I don't understand why you would need to do a logical OR to get the file handle in to the hash.

Thanks again for the help, and sorry for all the questions.

-Zipper

 

[KevinADC]

in english it means equals/or:

my $blah = 'blah';
my $foo = 'foo';
my $bar = $foo ||= $blah;
print $bar;
print "\n$foo";

the above prints "foo" but if $foo was not defined it would assign the value of $blah to $foo and then to $bar and print "blah". $blah is a fall through (or default) value.

Miller has a do{} block as the fall through assingment value instead of a scalar but it still works the same.

It doesn't need three possible values to work, two will work:

$bar ||= $foo;

In the above, if $bar has a value it will just keep that value, but if not it will assign the value of $foo to $bar. Its the same as:

if ($bar) {
    $bar = $bar;
}
else {
    $bar = $foo;
}

and the same as:

$bar = $bar ? $bar : $foo;

------------------------------------------
- Kevin, perl coder unexceptional!

 

[MillerH]

Thank you Kevin,

Basically, it's called a lazy initializer. The file handle for each $id is not created until it is actually seen. But once it is seen, it's cached in the hash %fh_of and does not need to be recreated.

As the documentation says, it is literally equivalent to:

    [url=http://perldoc.perl.org/functions/my.html][black][b]my[/b][/black][/url] [blue]$fh[/blue] = [blue]$fh_of[/blue][red]{[/red][blue]$id[/blue][red]}[/red] = [blue]$fh_of[/blue][red]{[/red][blue]$id[/blue][red]}[/red] || [url=http://perldoc.perl.org/functions/do.html][black][b]do[/b][/black][/url] [red]{[/red]
        [black][b]my[/b][/black] [blue]$filename[/blue] = [red]"[/red][purple][blue]$id[/blue].txt[/purple][red]"[/red][red];[/red]
        [url=http://perldoc.perl.org/functions/open.html][black][b]open[/b][/black][/url][red]([/red][black][b]my[/b][/black] [blue]$handle[/blue], [red]'[/red][purple]>[/purple][red]'[/red], [blue]$filename[/blue][red])[/red] or [url=http://perldoc.perl.org/functions/die.html][black][b]die[/b][/black][/url] [red]"[/red][purple]Can't open [blue]$filename[/blue]: [blue]$![/blue][/purple][red]"[/red][red];[/red]
        [blue]$handle[/blue][red];[/red]
    [red]}[/red][red];[/red]

As that link I gave you explained, almost any operator can be grouped with an assignment in such a way. We're just used to increment += and decrement -= only normally, since those come from C.

However, in this case, $fh_of{$id} is assigned to itself, or if it doesn't have a value, to the last statement of the do block.

And it's my habit to create a shortcut variable ($fh) in such a situation. Although in my specific example, it's kind of unnecessary since we only use it once.

- Miller