Remove duplicated lines with Perl


/ Published in: Perl
Save to your folder(s)

Name the source duplicate_remove.perl and open Terminal.app in the same directory, and write: perl duplicate_remove.perl


Copy this code and paste it in your HTML
  1. #!/usr/bin/perl
  2. # Usage in command line: perl duplicate_remove.perl <file-to-be-convertet>
  3. # Remove duplicated lines in text-files
  4.  
  5. my $origfile = $ARGV[0];
  6. my $outfile = "no_duplicates_" . $origfile;
  7. my %hTmp;
  8.  
  9. open (IN, "<$origfile") or die "Couldn't open input file: $!";
  10. open (OUT, ">$outfile") or die "Couldn't open output file: $!";
  11.  
  12. while (my $sLine = <IN>) {
  13. next if $sLine =~ m/^\s*$/; #remove empty lines
  14. #Without the above, still destroys empty lines except for the first one.
  15. print OUT $sLine unless ($hTmp{$sLine}++);
  16. }
  17. close OUT;
  18. close IN;
  19. print "The file $origfile is convertet! Look for $outfile in the same directory";

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.