Remove duplicates from a list


/ Published in: Django
Save to your folder(s)

Not only is it really really fast; it's also order preserving and supports an optional transform function


Copy this code and paste it in your HTML
  1. def remove_duplicates(seq, idfun=None):
  2. # order preserving
  3. if idfun is None:
  4. def idfun(x): return x
  5. seen = {}
  6. result = []
  7. for item in seq:
  8. marker = idfun(item)
  9. # in old Python versions:
  10. # if seen.has_key(marker)
  11. # but in new ones:
  12. if marker in seen: continue
  13. seen[marker] = 1
  14. result.append(item)
  15. return result
  16.  
  17.  
  18.  
  19. >>> a=list('ABeeE')
  20. >>> f5(a)
  21. ['A','B','e','E']
  22. >>> f5(a, lambda x: x.lower())
  23. ['A','B','e']

URL: http://www.peterbe.com/plog/uniqifiers-benchmark

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.