Best idea for output processing 1

[nfaber]

Hello All,

I need to generate an output file with this basic format:

ticket table:
location, priority, devType, startBacklog, opened, closed, endBacklog, total, changes, imBacklog, power, telco
Tier 1, critical, router, 0, 1,  1,  0, 1,  0, 0, 0,  1
Tier 2, critical, router, 0, 22, 21, 1, 22, 2, 1, 11, 8
Tier 2, warning,  switch, 0, 3,  3,  0, 3,  2, 0, 1,  0
Tier 2, normal,   router, 0, 3,  1,  2, 3,  0, 2, 0,  1
Total,        ,         , 0, 29,26,  3,29,  4, 3, 12,  10

Heres what I am going to do. I will be looping through a hash with this data structure (this is preliminary, not finalized yet:

$results{$logical_name} = ("$numberprgn, $assignment, $uis_elapsed_start_time, $close_time, $res_anal_code, $user_priority,	$severity_code, $type, $subtype, $uis_support_level, $uis_tier, $uis_fstatus");

Many if the fields are used earlier in my script. What I am interested in for this output file is the $uis_tier (Tier level in the output file), $severity_code (field to the right of Tier, critical, normal etc), $type (one more to the rigt, router, switch server etc.)

I need to build this file dynamically, as in there may be any number of entries for each column as in for $type there could be a router, switch, server, application etc, or just a router and switch. The same is true for each column, the file needs to be built up from left to right as in for each tier level we find, count the number of each severity code, then count the number for that severity code for each type and write all to the output file. The number of rows changes , but the number of columns stays the same.


I am planning on writing all the $uis_tier levels into an array then looping through to see how many different tier levels there are, then put each level in their own array (say @tier1, @tier2 etc.) then looping through the @tier1 array, for each type for that tier level write them into array etc.

At the same time (hopefully), the numbers coresponding to each tier=>criticality=>type will be counted for each column to fill in the numbers.

If anyone has done something similar and can think of a better way (code snippets, modues etc.) it would be appriciated.

Thanks,

Nick

@tier1
@tier2

I got a Biz Degree! How the h*ll did I get here?

 

[KevinADC]

If you have the time and a few dollars, get the CD, The Perl CD Bookshelf, it is invaluable.

Hopefully this tut on cpan will help you:

http://search.cpan.org/dist/perl/pod/perldsc.pod

it's a bit of a big pill to swallow at first, but stick with it and the power of references and annonymous data records will open a whole new world up for you if it hasn't already.

You will also want to aquaint yourself with Data:umper and/or Data:ump (easier to use IMHO) to print your records while learning and working with complex data records, they can be invaluable for debugging and learning.

 

[nfaber]

Thanks for the reply Kevin. I have been struggling trying understand references for some time now. I have read may tuts and books and have not "got" it yet.

I have one more week to get the project I am working on done or I am in deep dodo.

I guess I am looking for a start on the code I need to accomplish this and maybe finally understand how to use references, however I will re-read the material I have.



I got a Biz Degree! How the h*ll did I get here?

 

[nfaber]

What data structure lends itself to this? I am thinking an array of hashes and populate it like this:

foreach $tier (@tiers) {
	if ($tier = "Tier 1") {
		my @tier1 = (
       {severity_code => $severity_code,  type => $type},
		);
}

I got a Biz Degree! How the h*ll did I get here?

 

[KevinADC]

I might be tempted to start with something like this, a hash of hashes, and use an array only to store the totals for each of the three main columns. Based on your example from your first post it might look like this:

my %Tickets = (
   'Tier 1' => {
      priority => {
         normal   => {
            router => [],
            switch => [],
            server => [],
            application => []
         },
         warning  => {
            router => [],
            switch => [],
            server => [],
            application => []
         },
         critical => {
            router => [0, 1,  1,  0, 1,  0, 0, 0,  1],
            switch => [],
            server => [],
            application => []
         }
      }
   },
   'Tier 2' => {
      priority => {
         normal   => {
            router => [0, 3,  1,  2, 3,  0, 2, 0,  1],
            switch => [],
            server => [],
            application => []
         },
         warning  => {
            router => [],
            switch => [0, 3,  3,  0, 3,  2, 0, 1,  0],
            server => [],
            application => []
         },
         critical => {
            router => [0, 22, 21, 1, 22, 2, 1, 11, 8],
            switch => [],
            server => [],
            application => []
         }
      }
   }
);

The thing with your data is that its pretty well fixed as far as what data there is, it's not so dynamic that you never know whats coming, unless there are many more priorites and types than you suggest. But if you know all the priorities and types you can build a fixed structure like above and then populate the appropriate array with the totals. Or maybe one of the really advanced members of this forum like mikeevh or PaulTEG, or another, will have a better or easier suggestion.

 

[nfaber]

Thanks for the info and code Kevin. That is a good start. You say here:

The thing with your data is that its pretty well fixed as far as what data there is, it's not so dynamic that you never know whats coming, unless there are many more priorites and types than you suggest.

It really is not fixed. From my original post:

I need to build this file dynamically, as in there may be any number of entries for each column as in for $type there could be a router, switch, server, application etc, or just a router and switch. The same is true for each column, the file needs to be built up from left to right as in for each tier level we find, count the number of each severity code, then count the number for that severity code for each type and write all to the output file. The number of rows changes , but the number of columns stays the same.

Maybe some type of a foreach loop to build the hash of hashes would work.

I got a Biz Degree! How the h*ll did I get here?

 

[nfaber]

Kevin, when you lay it out that way it is easier to understand. I can just fill in the numbers I get from my calcualtions done earlier in my program with variables rather than the hard coded ones here:

         critical => {
            router => [0, 1,  1,  0, 1,  0, 0, 0,  1],
            switch => [],
            server => [],
            
         }

I could use that as:

         critical => {
            router => [$startBacklog, $opened, $closed, $endBacklog, etc.],
            switch => [],
            server => [],
            
         }

I got a Biz Degree! How the h*ll did I get here?

 

[KevinADC]

It really is not fixed. From my original post:

Yes, I understand that you will be building the data records dynamically, but you can start with a totally fixed data structure, at least I think you can. All you really seem to be doing is adding up those last 7 columns, but you know what the columns are. Your struture is fixed. This should make filling in the blanks a fairly straight forward task.

You will need to use a loop of some sort to fill in the data, that really goes without saying. Could be a while loop, a for or foreach loop, whatever seems appropriate for the input you are working with.

If the second and third columns don't have too many potential values I would maybe just create all the values at the very start and fill them in with data as necessary. But if there are many values for the second and third column you can create them as needed.

Kevin, when you lay it out that way it is easier to understand. I can just fill in the numbers I get from my calcualtions done earlier in my program with variables rather than the hard coded ones here:

Yes! Exactly. And since you are wanting a total at the end you can keep a running total stored in another variable right within the same top level hash: %Tickets.

my %Tickets = (
   'totals' => [0,0,0,0,0,0,0], 
   'Tier 1' => {
    ......
);

you add the totals into the appropriate array index as you go along.

 

[nfaber]

Kevin,

I'm still confused about how I use the hash of hashes. If I already have the data in scalars for the ticket numbers in the chart, why write them to a hash of hashes? Why not just write them directly to the output file?

I got a Biz Degree! How the h*ll did I get here?

 

[KevinADC]

Kevin,

I'm still confused about how I use the hash of hashes. If I already have the data in scalars for the ticket numbers in the chart, why write them to a hash of hashes? Why not just write them directly to the output file?

hehehe.... I don't know. You asked for suggestions, I gave you mine.

 

[nfaber]

Mike/Paul,

Any ideas from you guys?

Nick

I got a Biz Degree! How the h*ll did I get here?

 

[rharsh]

nfaber, what exactly are you looking for? I'm assuming you want a better way to organize, then output, the data. But you didn't want to use the data structure KevinADC suggested (which is similar to what I would have suggested.)

 

[nfaber]

rharsh,

I think I made an error with this post that I try to avoid, that is asking for to much info at once, and not having a clear idea what I am trying to get.

I have done a little research into the data structures involved with this (and Kevin explination helped alot) but I am looking for a way to easily create access and change the data structure (who isn't). What I am looking for I guess is a way to access the data as in:

$OutputRow = Tier{'Tier 1'}=>priority{critical}=>type{router} and then get the values for the columns in the chart for that row as in type{router}[0, 1, 1, 0, 1, 0, 0, 0, 1].

I assume I can add data the same way as in

Tier{'Tier 1'}=>priority{critical}=>type{router} = [0, 1, 1, 0, 1, 0, 0, 0, 1]

and I need a good way to loop through the hashes to when adding of accessing the data. I can get the access, add, loop pretty well from the web tuts and my books, however I am a little inexperienced with the right structure for my multidemensional DS.

I am not sure if a Hash of Hashes will allow me to access the data as I have described above.

I got a Biz Degree! How the h*ll did I get here?

 

[KevinADC]

Okay, let me see: say you have records like this being produced:

$results{$logical_name} = ("$numberprgn, $assignment, $uis_elapsed_start_time, $close_time, $res_anal_code, $user_priority,    $severity_code, $type, $subtype, $uis_support_level, $uis_tier, $uis_fstatus");

I don't know which scalars actually correspong to what data but I know the first three, so you can do something like this with the data (assumes open filehandle FILE):

my %Tickets = (); 
while(<FILE>){
   chomp; #if necessary
   # get you data into the list of scalars you posted
   # in your first post
   my @data = qw($numberprgn $assignment $uis_elapsed_start_time $close_time $res_anal_code $user_priority $subtype $uis_support_level  $uis_fstatus);
   for my $i (0..8) {
      $Tickets{$uis_tier}{$severity_code}{$type}[$i]+=$data[$i];
   }

that will build the hash of hashes and store the number values in an array corresponding to each possible combination of tier, severity, type. Now you have all this data you want to print to a file:

my %Tickets = (); 
while(<FILE>){
   chomp; #if necessary
   # get you data into the list of scalars you posted
   # in your first post
   my @data = qw($numberprgn $assignment $uis_elapsed_start_time $close_time $res_anal_code $user_priority $subtype $uis_support_level  $uis_fstatus);
   for my $i (0..8) {
      $Tickets{$uis_tier}{$severity_code}{$type}[$i]+=$data[$i];
   }
#finished building records
#print records to file
open (FILE,">records.txt") or die "$!";
flock(FILE, 2);
foreach my $tiers (keys %Tickets) {
	foreach my $severity (keys %{ $Tickets{$tiers} }) {
      foreach my $type (keys %{ $Tickets{$tiers}->{$severity} }) {
         print FILE "$tiers,$severity,$type," . join(',',@{ $Tickets{$tiers}->{$severity}->{$type} }) . "\n";
      }
   }
}
close(FILE);

so now your file should be like this:

Tier 1,critical,switch,3,4,0,1,4,0,3,0,2
Tier 1,critical,application,0,0,3,0,0,3,3,2,3
Tier 1,critical,server,0,2,2,0,0,1,0,0,0
Tier 1,normal,application,0,0,3,0,0,3,2,3,3
Tier 1,normal,server,0,1,0,1,0,0,2,0,0
Tier 1,normal,router,3,0,3,0,0,0,3,3,0
Tier 1,warning,application,3,6,3,6,0,0,0,3,0
Tier 1,warning,router,3,0,1,3,0,0,3,0,0
Tier 2,critical,switch,4,5,0,3,4,0,4,0,3
Tier 2,critical,application,0,0,3,0,0,3,3,2,3
etc
etc
etc

Now you can open the above file and rebuild the data structure at anytime:

my %Tickets = ();
open(FILE,"records.txt") or die "$!";
while(<FILE>){
   chomp;
   my ($tier,$severity,$type,@data) = split(/,/);
   $Tickets{$tier}{$severity}{$type} = [@data];
}

now you have some new data to update the records with:

my $user_tier = 'Tier 1';
my $severity = 'normal';
my $type= 'switch';
my @update = qw($numberprgn $assignment $uis_elapsed_start_time $close_time $res_anal_code $user_priority $subtype $uis_support_level  $uis_fstatus);
for my $i (0 .. $#update) {
   $Tickets{$user_tier}}{$severity}{$type}[$i]+=$update[$i];
}

then reopen the file for printing and reprint the data back to the file as shown above.

 

[nfaber]

Thanks much Kevin. Let me chew on that for a while until I understand.

Nick

I got a Biz Degree! How the h*ll did I get here?

 

[KevinADC]

You're welcome, I did spot an error or two in my code.

this part here:

while(<FILE>){
   chomp; #if necessary
   # get you data into the list of scalars you posted
   # in your first post
   my @data = qw($numberprgn $assignment $uis_elapsed_start_time $close_time $res_anal_code $user_priority $subtype $uis_support_level  $uis_fstatus);
   for my $i (0..8) {
      $Tickets{$uis_tier}{$severity_code}{$type}[$i]+=$data[$i];
   }
}
close(FILE);
#finished building records

obviously you need to end the while loop and close the file handle before printing the data to a file. And I see a syntax error here:

$Tickets{$user_tier}}{$severity}{$type}[$i]+=$update[$i];

there is an extra curly bracket:

$user_tier}}

 

[nfaber]

Kevin,

Ya, as I look through your code it is becoming clear to me what you are doing. I think I might finally get this multi-dimensional data structure monkey of my back! I'm really busy this week in a class, but I will get a chance to work on this this weekend. I attribute my beginning to get this to three things:

1) At this new job I am doing 90% perl as opposed to 50% before. I have been working on the same project since late November every day, programming, programming, programming ( I am even starting to dream about it).

2) It's amazing how much easier it is to understand code when the variables are variables you work with every day! Just seeing my DB queries in your code helps put it together.

3) Last and most important the skill and patience you have shown over the last few months helping me while I have been struggling with this. That goes for Paul and Mike as well.

Where do I send the check.

Nick

I got a Biz Degree! How the h*ll did I get here?