I am in need of some help. On a routine bases I have large database files that I need to extract data from. I then need to input the extracted data into new files.<br><br>The data is delimited, <br><br>Example:<br><br>****<br>Joe Smith<br>408-123-4567<br>Sample Co Inc.<br><br>(Followed by a multiple more lines)<br><br>****<br><br>The common delimiter between each set of data is ****<br>The data is then moved to a new file. I then continue to do this 100 or so more times until the file is empty. I have the ability to change the delimiter. The new files can also be named anything. The only requirement is they have a .tag extension.<br><br><br>My question: Does any know how I can automate this process through a script. I think SED would accomplish this, but I am not familiar enough with SED to execute it.<br><br>Any thoughts on automating this task would be helpful. <br><br>Danny.<br><A HREF="mailto:dannyd@aboveboardelectronics.com">dannyd@aboveboardelectronics.com</A><br>
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations wOOdy-Soft on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
SED / AWK
- Thread starter abovebrd
- Start date
Annihilannic
MIS
Danny,<br><br>It sounds like awk might be the tool for the job. You'll have to describe the output format you need more explicitly for a more detailed answer though... do you need it to be comma delimited rows? SQL insertion statements?<br><br>Annihilannic.
- Thread starter
- #3
Here is a sample of my source file :<br><br>cvr=none<br>tfn=408-573-5542<br>fll=<<EOF<br><br>ABC Company<br><br>00402932 IN 05/26/00 16.00 0.00 16.00 0.00 0.00 0.00<br>00403326 IN 05/31/00 197.32 0.00 197.32 0.00 0.00 0.00<br> ------------------------------------------------------------<br><br><br> 213.32 0.00 213.32 0.00 0.00 0.00<br>EOF<br><br><br><br>cvr=none<br>tfn=408-573-5542<br>fll=<<EOF<br><br>XYZ Company<br><br>00405461 IN 06/13/00 144.29 0.00 144.29 0.00 0.00 0.00<br> ------------------------------------------------------------<br><br><br> 144.29 0.00 144.29 0.00 0.00 0.00<br>EOF<br><br><br><br>I need to grab all of the information between cvr=none and<br>EOF. I then need to move that information to a new file. I would then need to repeat the process on the next occurence and so on until all occurences are moved to new files. <br>My source file has around 30,000 lines. There would also be around 500 files generated from this process.<br><br>If you have any ideas I would love to here them<br><br>Danny <br><br> <br> <p>Danny Daniels<br><a href=mailto:dannyd@aboveboardelectronics.com>dannyd@aboveboardelectronics.com</a><br><a href= > </a><br>
Annihilannic
MIS
This awk script does the trick:<br>-------------------- 8< -----------------------<br>#!/usr/bin/awk -f<br>BEGIN { <br> FILEINDEX=0 <br> OUTFILENAME="record." FILEINDEX<br>}<br>/^cvr=/ {<br> close(OUTFILENAME)<br> FILEINDEX++<br> OUTFILENAME="record." FILEINDEX<br>}<br>{ <br> print $0 > OUTFILENAME<br>}<br>-------------------- 8< -----------------------<br><br>Basically every time it encounters a line beginning with 'cvr=' it opens a new output filename.<br><br>I tried to do it using '\nEOF\n' as a record separator, but it didn't seem to work... if anyone can tell me why I'd appreciate it!<br><br>Annihilannic.
- Thread starter
- #6
Annihilannic,<br><br>I can change the syntax of the record separator if that helps. <br><br>AndyBo,<br><br>Perl is not loaded on this server. But installing perl is not out of the question ?<br><br>Danny<br><A HREF="mailto:dannyd@aboveboardelectronics.com">dannyd@aboveboardelectronics.com</A><br><br> <p>Danny Daniels<br><a href=mailto:dannyd@aboveboardelectronics.com>dannyd@aboveboardelectronics.com</a><br><a href= > </a><br>
Annihilannic: "\nEOF\n" probably wouldn't work because what you've actually got is "^EOF\n". ie, from beginning of the line ("^"
look for "EOF" followed by a newline ("\n".<br><br>Danny: Installing a full blown perl might be overkill to solve a single problem. I'd go with the "awk" solution first as awk will already installed. If you're having problems getting the awk to run, post back and we'll get into some perl hacking <p> <br><a href=mailto: > </a><br><a href= > </a><br>--<br>
0 1 - Just my two bits
look for "EOF" followed by a newline ("\n".<br><br>Danny: Installing a full blown perl might be overkill to solve a single problem. I'd go with the "awk" solution first as awk will already installed. If you're having problems getting the awk to run, post back and we'll get into some perl hacking <p> <br><a href=mailto: > </a><br><a href= > </a><br>--<br>0 1 - Just my two bits
- Thread starter
- #8
Thanks AndyBo, Annihilannic: <br><br>I ran the awk script earlier today and it seemed to do to trick. If fact it worked great !!!!<br><br>Thanks guys,<br><br>Just one more question:<br>Can you recommend a good good source for learing AWK. <br>Maybe a good book ?<br><br>Danny <br><A HREF="mailto:dannyd@aboveboardelectronics.com">dannyd@aboveboardelectronics.com</A><br> <br> <p>Danny Daniels<br><a href=mailto:dannyd@aboveboardelectronics.com>dannyd@aboveboardelectronics.com</a><br><a href= > </a><br>
I would say the "classic" Sed & Awk reference is "Sed & Awk" from O'Reilly. There are condensed highlights taken from this book in the "Unix in a Nutshell" and "Unix Power Tools" books also from O'Reilly. It's probably better to go for the original, though, if you want to really get down and learn some Awk. <p> <br><a href=mailto: > </a><br><a href= > </a><br>--<br>
0 1 - Just my two bits
0 1 - Just my two bits
- Status
Similar threads
- Locked
- Question