Advanced Search

Remove duplicated lines with Perl

/ Published in: Perl
Save to your folder(s)
Name the source duplicate_remove.perl and open Terminal.app in the same directory, and write: perl duplicate_remove.perl

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. #!/usr/bin/perl
  2. # Usage in command line: perl duplicate_remove.perl <file-to-be-convertet>
  3. # Remove duplicated lines in text-files
  4.  
  5. my $origfile = $ARGV[0];
  6. my $outfile  = "no_duplicates_" . $origfile; 
  7. my %hTmp;
  8.  
  9. open (IN, "<$origfile")  or die "Couldn't open input file: $!"; 
  10. open (OUT, ">$outfile") or die "Couldn't open output file: $!"; 
  11.  
  12. while (my $sLine = <IN>) {
  13.   next if $sLine =~ m/^\s*$/;  #remove empty lines
  14.                                #Without the above, still destroys empty lines except for the first one.
  15.   print OUT $sLine unless ($hTmp{$sLine}++);
  16. }
  17. close OUT;
  18. close IN;
  19. print "The file $origfile is convertet! Look for $outfile in the same directory";

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.