Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations Chriss Miller on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
How to check for duplicate entries in a file and remove the duplicate?
- Thread starter JTan
- Start date
TrojanWarBlade
Programmer
sort will, of course, resequence your file.
If this is not a problem then all well and good but if it is a problem then you have two possible solutions.
1) write a script to remove dupes using an associative array of some kind to see if the record to be output has been seen before.
2) use "seq" to create an index and "join" it to the end of each record in the first file. Then use "sort" to resequence AND remove dupes. Next sort again on the index we created (to sort back to the original sequence). Finally "cut" the index off again. Hey presto, your file is de-duped but not resequenced.
Trojan.
If this is not a problem then all well and good but if it is a problem then you have two possible solutions.
1) write a script to remove dupes using an associative array of some kind to see if the record to be output has been seen before.
2) use "seq" to create an index and "join" it to the end of each record in the first file. Then use "sort" to resequence AND remove dupes. Next sort again on the index we created (to sort back to the original sequence). Finally "cut" the index off again. Hey presto, your file is de-duped but not resequenced.
Trojan.
Hi,
see thread
see thread
Code:
awk '!a[$1]++' /path/to/your_file- Status
Similar threads
- Locked
- Question