deleting a pattern in all files in a directory

[amjadcsu]

hi folks
I have a directory called xml_logs which records real time conversation on IRC servers.
This directory consists of xml files
Each file has a format <!CDATA[chat data]]> in it
I need a script to remove this pattern from each file in directory and
the result should be
<chatdata>

i have tried to use sed but sed works on list on files how to make it work on directory in a bash script. any help
thanks

 

[KevinADC]

do you want to use a shell script or perl?

 

[amjadcsu]

well i would like to use both .
whichever is easy

 

[icrf]

i have tried to use sed but sed works on list on files how to make it work on directory in a bash script. any help

Any one-liner I come up with in perl is liable to have the same limitation. *nix shells expand wildcards into lists, so if you want it to work on a directory, instead of passing /path go with /path/* and you get a list of all files in /path

- Andrew
Text::Highlight - A language-neutral syntax highlighting module in Perl
also on SourceForge including demo

 

[derekpc]

You didn't say whether everything between <!CDATA[ and ]]> is on a single line, but, if so, something like the below should work. If not you would have to handle matching across multiple lines. I am sure there is a cleaner way, but it should work substituting ./xml_logs with whatever the correct path is to the logs directory and /tmp with an appropriate temp directory.

#!/usr/bin/perl
use strict;
use warnings;
use File::Copy;
opendir TEST, "./xml_logs" || die "opendir failed: $!\n";
my @files = readdir(TEST);
closedir TEST;
for my $file ( @files ) {
    next unless ( -f "./xml_logs/$file" );
    open(OLD, "< ./xml_logs/$file") || die "failed to open $file because: $!\n";
    open OUTFILE, "> /tmp/workfile" || die "failed to open /tmp/workfile because: $!\n";
    while (<OLD>) {
	s/!CDATA\[(chat data)\]\]/$1/ig; #
	print OUTFILE;
    }
    close OLD;
    close OUTFILE;
    copy("/tmp/workfile", "./xml_logs/$file") || die "Copy failed: $!\n";
}

Derek

 

[KevinADC]

Dereks code just needs a small tweak:

s/!CDATA\[(chat data)\]\]/<$1>/ig; #

and if you really want "chatdata" instead of "chat data" you will have to assign $1 to a temp variable and modify the variable to remove the space.

 

[KevinADC]

or you could:

s/!CDATA\[(chat data)\]\]/<chatdata>/ig; #

or if you wanted the space between the words:

s/!CDATA\[(chat data)\]\]/<chat data>/ig; #

 

[amjadcsu]

ok
the problem is that temp file is created but it is blank.
As such nothing changes in original xml files

 

[KevinADC]

this is very similar but maybe give it a try:

#!/usr/bin/perl
use strict;
use warnings;
my $start_dir = 'path/to/xml_logs';
chdir($start_dir) or die "Can't chdir $start_dir: $!";
my @files = <*.xml>;
my @errors = ();
foreach my $file ( @files ) {
    open(OLD, "<$file") or push @errors, "failed to open $file: $!";
    open(TEMP, ">temp.txt") or push @errors, "failed to create temp file: $!";
    while (my $line = <OLD>) {
       $line =~ s/!CDATA\[(chat data)\]\]/<chatdata>/ig;
       print TEMP $line;
    }
    close OLD;
    close TEMP;
    rename('temp.txt', $file) or push @errors, "Rename failed: $!";
}
if (@errors) {
   print "$_\n" for @errors;
}
else {
   print "No errors reported";
}

 

[rharsh]

Yeah, I would probably go with ircf's sugggestion.

This might work for you:

perl -pi.bak -e 's/<!CDATA[chat data]]>/<chatdata>/gi' /path/to/files/*.xml

Or, if you don't want the backup files, run it similar to:

perl -pi -e ...

And, of course, run that on copies of the files first - make sure it does what you want.

 

[icrf]

Yeah, same works for sed if you already had a file-by-file sed command written.

- Andrew
Text::Highlight - A language-neutral syntax highlighting module in Perl
also on SourceForge including demo

 

[rharsh]

That's true. Although, it depends on the version of sed. The crappy one installed on all of our Solaris 9 boxes doesn't have in-place editing! It's crap I tell you.

 

[icrf]

What do I know? I've never used sed. I hear rumors it does the same kind of thing as CLI perl, but people that voluntarily skip out on perl are crazy, I don't trust them.

- Andrew
Text::Highlight - A language-neutral syntax highlighting module in Perl
also on SourceForge including demo

 

[amjadcsu]

ok
now instead of chat data if i have something dyanmically created by chat bots

It could be any damn string and not chat data.
Also the format is

<![CDATA[#IRC]]>

where IRC can be replace by any string
I want the output to be

<#IRC>

 

[KevinADC]

sticking with rharsh's one liner, try this:

perl -pi.bak -e 's/<![CDATA[([^\]]*)]]>/<$1>/gi' /path/to/files/*.xml

or you may have to escape the square brackets:

perl -pi.bak -e 's/<!\[CDATA\[([^\]]*)\]\]>/<$1>/gi' /path/to/files/*.xml

 

[amjadcsu]

ok
It does not do anything.
the perl liner does not change anything in original file

 

[KevinADC]

well, You can retry this code:

#!/usr/bin/perl
use strict;
use warnings;
my $start_dir = 'path/to/xml_logs';
chdir($start_dir) or die "Can't chdir $start_dir: $!";
my @files = <*.xml>;
my @errors = ();
foreach my $file ( @files ) {
    open(OLD, "<$file") or push @errors, "failed to open $file: $!";
    open(TEMP, ">temp.txt") or push @errors, "failed to create temp file: $!";
    while (my $line = <OLD>) {
       $line =~ s/<!\[CDATA\[([^\]]*)\]\]>/<$1>/ig;
       print TEMP $line;
    }
    close OLD;
    close TEMP;
    rename('temp.txt', $file) or push @errors, "Rename failed: $!";
}
if (@errors) {
   print "$_\n" for @errors;
}
else {
   print "No errors reported";
}

make sure the path to your xml_logs directory is correct:

my $start_dir = 'path/to/xml_logs';

 

[amjadcsu]

Ok thanks
Now i want to do something more

like
if i have

<![CDATA[#IRC]]>

i want only

IRC
thanks

 

[KevinADC]

well, what have you tried?

 

[amjadcsu]

i tried the script
$line =~ s/<!\[CDATA\[([^\]]*)\]\]>/<$1>/ig;
not the perl one liner