How to tell how much memory an array/hash uses? 4

[lcs01]

One of my company's perl code uses lots of memories. Sometimes it may use up all the memory our live server has. This problems is probably caused by some arrays and/or hashes inside the code. These arrays/hashes can have more than 10 million data members. I have modified the codes and made the performance a bit better. But I want to find out how much memory those arrays/hashes are actually using during the run before/after my fix.

Can someone tell me how to do this?

In addition, I also want to know how much memory the whole program is using during the run. I know one way is using 'top' command. But each run may take quite some time to finish. So it is hard to keep my eye on the 'top' output. I suppose there must be a better way to monitor this?

Thank you very much for your help.

 

[mbrooks]

This is what you want.

http://search.cpan.org/~glennwood/Benchmark-Harness-1.07/

M. Brooks

http://mbrooks.info

 

[lcs01]

Thank you, mbrooks.

Could you please show me some sample codes? I looked at the site you recommended, but could not find anything about how to use the module to read. The discussion forum there seems down.

Thanks again.

 

[KevinADC]

just another suggestions:

http://search.cpan.org/~dsugal/Devel-Size-0.64/Size.pm

- Kevin, perl coder unexceptional!

 

[lcs01]

Thank you, Kevin,

I tried Size.pm. But the results did not make much sense to me. Here is a piece of my code:

use Devel::Size qw(size total_size);

#
# some implementation here
# ......
# ......
my $sql1 = "SELECT column1 FROM mytable";
#my $sql2 = "SELECT * FROM mytable"; # mytable has more than  80 columns
my $sth = $dbh->prepare($sql1);
$sth->execute;
my %data;
while (my $rowData = $sth->fetchrow_hashref) {
  $data{$$rowData{'id'} = $rowData;
}
$sth->finish;

my $size4DataHash = size(\%data);
print "\$size4DataHash = $size4DataHash\n";
#
# some implementation here
# ......
# ......

The value of $size4DataHash is the same for either $sql1 or $sql2 is used. Why?

Again, thank you for your help again.

 

[MillerH]

You need to use the total_size function instead of size. The size function only gives you the memory usage of the first laywer, in this case a hash reference. I'd expect that this would be a constant value, so if you want not to get the total size of the complex data structure, you'll have to use the "total_size" function. Read the documentaiton some more though, as it reports that there is a large memory footprint for this operation itself. But it will at least give you a feel for how much memory each structure takes up.

 

[lcs01]

Thank you, Miller and Kevin!

After I used total_size, the results turned out exciting. The code after my modification could save up to 92% of the memory!!

Again, thank you both!!