×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Help with an awk script to filter tables

Help with an awk script to filter tables

Help with an awk script to filter tables

(OP)
Dear Forum,

I need to filter text files according to some measured and recorded values (sensor). The files can be large and the recorded data points could differ in length (NF is variable). I have worked out a solution in awk (see below) but it is very messy and not flexible. I would appreciate your help for a better, cleaner and more robust way to solve my problem?

A typical input file would look like this:

CODE -->

ID_AB1\tsensor01(1.0),sensor02(0.6),sensor03(0.5),sensor04(0.45)
ID_AB2\tsensor01(1.0),sensor02(0.95),sensor03(0.90),sensor04(0.80)
ID_AC3\tsensor01(1.0),sensor02(1.0),sensor03(1.0),sensor04(1.0)
ID_AD1\tsensor01(0.9),sensor02(0.6)
ID_BA2
... 

At the end I would like an output file like the following:

CODE -->

ID_AB1;sensor01;low;low;low
ID_AB2;sensor01;sensor02;sensor03;low
ID_AC3;sensor01;sensor02;sensor03;sensor04
ID_AD1;sensor01;low;na;na
ID_BA2;na;na;na;na 

Sensors with a low value (e.g. PF=0.8) should be replaced as "low". The sensor signal is decreasing. It is possible that consecutive sensors have an equal value but a sensor has never a high signal to the one before. It is, however, possible that some sensor are missing and this needs to considered (e.g. na in the output)

My solution work and I could extend it to cover more possible cases but it is confusing and not flexible.

CODE --> awk

sed 's/(/,/g' out.singnal | sed 's/)//g' | awk -F"\t|," -v PF=0.8 '{
  printf "%s;", $1
  if(NF==9 && $3>=PF && $5>=PF && $7>=PF && $9>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,$8
  else if(NF==9 && $3>=PF && $5>=PF && $7>=PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"low"
  else if(NF==9 && $3>=PF && $5>=PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","low"
  else if(NF==9 && $3>=PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","low"
  else if(NF==9 && $3<PF && $5<PF && $7<PF && $9<PF)
          printf " %s; %s; %s; %s\n", "low","low","low","low"
  else if(NF==7 && $3>=PF && $5>=PF && $7>=PF)
          printf " %s; %s; %s; %s\n", $2,$4,$6,"na"
  else if(NF==7 && $3>=PF && $5>=PF && $7<PF)
          printf " %s; %s; %s; %s\n", $2,$4,"low","na"
  else if(NF==7 && $3>=PF && $5<PF && $7<PF)
          printf " %s; %s; %s; %s\n", $2,"low","low","na"
  else if(NF==7 && $3<PF && $5<PF && $7<PF)
          printf " %s; %s; %s; %s\n","low","low","low","na"
 }' 

Are there any awk wizards able to help me to improve my script? Thanks a lot!

RE: Help with an awk script to filter tables

Hi Lobar17,

Here is my attempt to simplify the code:

CODE

# Run:
#   awk -f lobar17.awk lobar17.txt

BEGIN {
  # field separator must be: \t (tab character) or comma
  FS = "  |,"
  LOW_VALUE = 0.8
} 

{ 
  id = $1
  s1 = sensor_value($2)
  s2 = sensor_value($3)
  s3 = sensor_value($4)
  s4 = sensor_value($5)
  printf("%s;%s;%s;%s;%s\n", id, s1, s2, s3, s4)
}

# -- fuctions --
function sensor_value(fld) {
  result = "nothing"
  if (match(fld, /(sensor[0-9][0-9])\((.+)\)/, match_list)) {
    sensor_name = match_list[1]
    num_val = match_list[2]
    if (num_val <= LOW_VALUE) {
      result = "low"
    }
    else {
      result = sensor_name
    }
  }
  else {
    result = "na"
  }
  return result
} 

Output:

CODE

$ awk -f lobar17.awk lobar17.txt
ID_AB1;sensor01;low;low;low
ID_AB2;sensor01;sensor02;sensor03;low
ID_AC3;sensor01;sensor02;sensor03;sensor04
ID_AD1;sensor01;low;na;na
ID_BA2;na;na;na;na 

RE: Help with an awk script to filter tables

the function sensor_value(fld) could be simplified:

CODE

function sensor_value(fld) {
  result = "na"
  if (match(fld, /(sensor[0-9][0-9])\((.+)\)/, match_list)) {
    sensor_name = match_list[1]
    num_val = match_list[2]
    if (num_val <= LOW_VALUE) {
      result = "low"
    }
    else {
      result = sensor_name
    }
  }
  return result
} 

RE: Help with an awk script to filter tables

(OP)
Hi mikrom,

Thanks a million - it works just perfect !!!

RE: Help with an awk script to filter tables

Hi Lobart17,

You're welcome!

RE: Help with an awk script to filter tables

Here's a different awk recipe:

CODE --> awk

awk -F"[\t]|[(]|[)][,]?" '{
    for (i=1;i<=NF;i++)
        x[$1] = (i==1 ? $1 : (i%2 ? x[$1] ";" ($i > 0.8 ? $(i-1) : "low") : x[$1]))
    n=split(x[$1],a,";")
    for (j=n;j<5;j++) x[$1]=x[$1]";na"
} END {for(i in x) print x[i]}' file 

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close