INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

awk multiple file processing

awk multiple file processing

(OP)
Hi!

Could you please help to modify the following awk code (script) to shorten it. This code goes through different files and calculates the mean and standard deviation of the last value in the second column. Now it is done in ``stupid'' way, i.e., if I want to add one more data set I would need to change a parameter N everywhere and add manually corresponding file name. Furthermore, adding different data corresponding to a parameter w (omega) would require adding one more block of the code. Though, since all these blocks have a similar structure I wounder if it is possible to make it easier with awk, i.e., have an array of different w (omega) I would like to process, and a parameter N corresponding to a number of files I need to process.

Thank you for your help and time.

CODE

<html>
  <head>
    <title>Page</title>
  </head>
  <body>
awk    '{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
    print 2 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=2/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=2/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=2/OUTPUT_ED/pre > ./DATA/errors_tus+.data

awk    '{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
    print 3 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=3/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=3/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=3/OUTPUT_ED/pre >> ./DATA/errors_tus+.data

awk    '{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
    print 4 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=4/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=4/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=4/OUTPUT_ED/pre >> ./DATA/errors_tus+.data

awk    '{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
    print 5 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=5/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=5/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=5/OUTPUT_ED/pre >> ./DATA/errors_tus+.data

awk    '{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
    print 6 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=6/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=6/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=6/OUTPUT_ED/pre >> ./DATA/errors_tus+.data
  </body>
</html>

RE: awk multiple file processing

How about wrapping it in a little shell code:

CODE

# empty the output file
> ./DATA/errors_tus+.data

for w in 2 3 4 5 6
do
    awk '
        { Nu[f]=0; Nu[f] = $2 }
        END {
            N=3;
            mu=0;
            for(i=1;i<=N;i++) { mu += Nu[i] }
            mu /= N;
            sigma=0;
            for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2}
            sigma = sqrt(sigma/N);
            print w " " mu " " sigma
        }
    ' w=$w f=1 ./_TUS+/N=4096/w=$w/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=$w/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=$w/OUTPUT_ED/pre >> ./DATA/errors_tus+.data
done

Annihilannic
tgmlify - code syntax highlighting for your tek-tips posts

RE: awk multiple file processing

(OP)
Thank you very much, Annihilannic!
Actually I did the same (shell+awk) recently when I realized that this is the easiest way to do it.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close