Advanced Search

Return to Snippet

Revision: 49978
at August 4, 2011 09:17 by ksaver
Updated Code 
# Bash one-liner for find duplicate files
# ksaver, Aug 2011
# http://www.commandlinefu.com/commands/view/8958/find-duplicate-files-using-sha1-hash
# Public Domain Code
# Updated with some nice changes, now is smaller and faster... :-)

for i in $(find . -type f -exec sha1 -r {} \+ |tee .hashes.tmp |awk '{print $1}' |sort |uniq -d); do grep $i .hashes.tmp; echo; done;

Revision: 49977
at August 3, 2011 19:52 by ksaver
Initial Code 
# Bash one-liner for find duplicate files
# ksaver, Aug 2011
# http://www.commandlinefu.com/commands/view/8958/find-duplicate-files-using-sha1-hash
# Public Domain Code

for i in $(find . -type f -exec sha1 -r {} \; |tee .hashes.tmp |awk '{print $1}' |sort |uniq -c |awk '{print $1, $2}'|grep -v "^1"|sort -rn |awk '{print $2}'); do grep $i .hashes.tmp; echo; done;
Initial URL 
http://www.commandlinefu.com/commands/view/8958/find-duplicate-files-using-sha1-hash
Initial Description 
Output Example:

d65bfef64a5fc9f7dbf9d35d80a2e1ed218c75d2 ./tmp1/12414.txt


d65bfef64a5fc9f7dbf9d35d80a2e1ed218c75d2 ./tmp2/2012.txt

d65bfef64a5fc9f7dbf9d35d80a2e1ed218c75d2 ./tmp1/3153.txt


dd07cec149e7c5929d6e9a0618de7114d50b34b0 ./tmp1/10064.txt

dd07cec149e7c5929d6e9a0618de7114d50b34b0 ./tmp2/30901.txt


d9bc21587f94d7a138bddf41cfa4e92a04cf9c54 ./tmp1/36.txt

d9bc21587f94d7a138bddf41cfa4e92a04cf9c54 ./tmp1/83.txt

[...]
Initial Title 
Find duplicate files, using sha1 hash
Initial Tags 
find
Initial Language 
Bash