×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Find all files in a directory and process only ones that have a specific line, skip if does not

Find all files in a directory and process only ones that have a specific line, skip if does not

Find all files in a directory and process only ones that have a specific line, skip if does not

(OP)
Need help, I’m new to python,
I wrote a script that should find all files in a directory and process only ones that have a specific line and skip the ones that do not have the line. The specific line is ‘," Run Time’ it fails to process ONLY files I need, it processes all files.

All Lines to find:

‘," Run Time’
‘,” Start Time’
‘,” End Time’
‘Test_ID:’
‘Test Program Name:’
‘Product:’
Also,
Lines 1, 2 and 3 are repeating lines and I need them all,
Lines 4, 5 and 6 also repeating but I need to capture them only ones.
Here is the script I have:

CODE --> python

import os

runtime_l = ',"  Run  Time'
start_tm  = ',"  Start Time'
end_tm    = ',"  End  Time'
test_ID   = ' Host Name: '
program_n = 'Test Program Name:'
prod_n    = 'Product:'

given_path = 'C:\\02\\en15\\TST'
for filename in os.listdir(given_path):
    filepath = os.path.join(given_path, filename)
    if os.path.isfile(filepath):
        print("File Name:   ", filename) 
        print("File Name\\Path:", filepath) 
        with open(filepath) as mfile:        
            for line in mfile:
                if runtime_l in line:
                    # do something with the line
                    print(line)
                    
                if start_tm in line:
                    # do something with the line
                    print(line)  

                if end_tm in line:
                    # do something with the line
                    print(line) 
                    
                if test_ID in line:
                    # do something with the line
                    print (line)
        
                if program_n in line:
                    # do something with the line
                    print (line)
        
                if prod_n in line:
                    # do something with the line
                    print (line)
                else:                    
                    continue 

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

Hi

Hard to tell without a sample file what would be the best way, but generally there would be 2 cases :
  • If the line containing runtime_l always occurs before the other lines, then use a flag : 1) initialize it to false; 2) switch it to true when runtime_l is met; 3) when encountering another line and the flag is false, stop processing the file.
  • If the line containing runtime_l may occur anywhere in the file, then loop over the file twice : if the runtime_l was found, close the file and start the reading loop again looking only for the rest of lines.

Feherke.
feherke.github.io

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

(OP)
I really appreciate your suggestion but I do not know what that means, I'm new to Python.
the File can be anything with the 6 lines inserted randomly.

The two questions I need help with are:
1. How I can test a file to make sure it has a line "Run Time", process it if it has the line, and skip the rest of the files in a directory?
2. how to print only first match and ignore the rest of the matched lines?
for example, I'm looking for a line in a file that has this string "Test Program Name:", the file can have 2 to 5 or even more lines with this string.
I need to find only the first matched line that has the string 'Test Program Name:' and if found I want the code to print the line and start looking for the next variable to match that is '‘Test_ID:'.

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

Hi Tester_V,

I'm trying to answer your two questions:

Quote (Tester_V)


1. How I can test a file to make sure it has a line "Run Time", process it if it has the line, and skip the rest of the files in a directory?
2. how to print only first match and ignore the rest of the matched lines?


1. You could select for processing only those files which contain "Run Time".
Look at the command

CODE

find $PWD -name "*.txt" | xargs grep -l "Run Time" 
used with popen() in my example below

2. You can mark when you have found one of the strings.
See the usage of the dictionary strings_found in my example below

Here is the example:
I created some files in a directory tree

CODE

.
├── dirname_01
│   └── file_04.txt
├── file_01.txt
├── file_02.txt
└── file_03.txt 
The files file_01.txt, file_03.txt and file_04,txt contain "Run Time".

tester_v.py

CODE

import os

def process_file(file_path):
  strings_found = {
    "Run Time":False,
    "Start Time":False,
    "End Time":False,
    "Test_ID":False,
    "Test Program Name":False,
    "Product":False
  }
  print("*** Processing %s:" % file_path)
  txt_file = open(file_path, "r")
  for line in txt_file:
    for key in strings_found.keys():
      if key in line and not strings_found[key]:
        print(line.rstrip())
        strings_found[key] = True
  #
  txt_file.close
  print("*** Done.\n")

# -- main program----
if __name__ == "__main__":
  cmd = 'find $PWD -name "*.txt" | xargs grep -l "Run Time"'
  file_names = os.popen(cmd)
  for file_name in file_names:
    process_file(file_name.rstrip()) 

Running and output:

CODE

$ python3 tester_v.py
*** Processing /home/mikrom/Work/Python3/process_files/file_01.txt:
," Run Time
,” Start Time
,” End Time
Test_ID:
Test Program Name:
Product:
*** Done.

*** Processing /home/mikrom/Work/Python3/process_files/dirname_01/file_04.txt:
," Run Time
,” Start Time
,” End Time
Test_ID:
Test Program Name:
Product:
*** Done.

*** Processing /home/mikrom/Work/Python3/process_files/file_03.txt:
," Run Time
,” Start Time
,” End Time
Test_ID:
Test Program Name:
Product:
*** Done. 

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

(OP)
Thank you! The code you wrote looks amazing!
And I'm sure it works great but it is too complicated.
Is there any other way? a simpler way to do this?
I need to incorporate your snippet into my code and I cannot do that, it is too complicated.
I really appreciate your help, thank you again!

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

And what is in your opinion complicated ?
Using the command with find and grep to select only those files which contain the strings you want to process, or using the dictionary to mark the strings found in file ?
The best way would be, if you take the program apart, step by step and print some variables to see how it works, then you will see that it is not complicated.

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

(OP)
You guys trying to help and I see it and appreciate it but I'm not on the same level as you in Python.
That is why obvious things for you look very complicated to me and I'm trying to do everything in very simple way.
Here is how I would try to test a file if it has a "Run Time" line in it and it works, kind of.
In a directory 'C:\\02\\en15\\TST' I'm skinning I have 4 files:
Debug_1.log
Debug_2.log
Debug_3_NO_Run Time
Debug_3_NO_Run Time
First two Debug log files have the "Run Time" line and the last two do not have the lines. When I execute my script with "else" block disabled, the script prints correctly:

<_io.TextIOWrapper name='C:\\02\\en15\\TST\\Debug_1.log' mode='r' encoding='cp1252'>
<_io.TextIOWrapper name='C:\\02\\en15\\TST\\Debug_2.log' mode='r' encoding='cp1252'>
>>>

When I enable the "else" block the script produces an error for some reason.
else:
^
IndentationError: expected an indented block
>>>



CODE --> python

import os

runtime_l = ',"  Run  Time'

given_path = 'C:\\02\\en15\\TST'
for filename in os.listdir(given_path):
    filepath = os.path.join(given_path, filename)
    if os.path.isfile(filepath):
        #print("File Name:   ", filename) 
        #print("File Name\\Path:", filepath) 
        with open(filepath) as mfile:        
            for line in mfile:
                if runtime_l in line:
                    # do something with the line
                    print(mfile)
                 
                #else:                    
                #    print(mfile) 

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

Hi Tester_V,
The code you posted works with me with else too. As your error message said, you must have an IndentationError near of the else. Maybe you mixed tabs and spaces for indentation.

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

(OP)
You R right! Thank you mikrom!

RE: Find all files in a directory and process only ones that have a specific line, skip if does not

(OP)
To mikrom, thank you!

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close