How to find duplicate files safely

Phone backups, "copy (1)" downloads and the same folder imported twice: find every exact duplicate, keep one copy of each, and never delete the wrong one.

How exact duplicate finders work

Two files are duplicates when their contents are identical, whatever their names or dates. The fast way to find them:

1. Group by size. A file with a unique size can't have a duplicate, so most files are never read.
2. Hash the rest (SHA-256). Same hash means same content.
3. Skip empty files. They all "match" each other.

This finds exact copies only. A photo that was resized, edited or re-saved is a different file; for those you need a similar image finder (see Czkawka below).

Option 1: Windows PowerShell (built in)

Open PowerShell, go to the folder (cd $HOME\Pictures) and paste. It only lists the duplicate groups, one blank line between groups, and changes nothing:

Get-ChildItem -Recurse -File | Where-Object Length -gt 0 |
  Group-Object Length | Where-Object Count -gt 1 |
  ForEach-Object { $_.Group | Get-FileHash -Algorithm SHA256 } |
  Group-Object Hash | Where-Object Count -gt 1 |
  ForEach-Object { $_.Group.Path; '' }

Tested in PowerShell 7.4. On a test folder it printed the two copies of report.pdf and the three copies of IMG_1001.jpg, and ignored two different files of the same size and two empty files.

Option 2: macOS and Linux terminal (built in)

In the folder you want to check (Linux; on macOS replace sha256sum with shasum -a 256):

find . -type f -size +0c -exec sha256sum {} + | sort | awk '
  $1 == prev { if (!shown) print prevline; print; shown = 1; next }
  { if (shown) print ""; prev = $1; prevline = $0; shown = 0 }'

Output, one group per block (same hash = same content):

a5e10341fc2fa1cd28fc7036e7d23d46e04d66948c80d243ad425baf05cc7fe0  ./report copy.pdf
a5e10341fc2fa1cd28fc7036e7d23d46e04d66948c80d243ad425baf05cc7fe0  ./report.pdf

f318e36417590c104b60ea801214aff6fe53407e25c8439e392054e77b59d897  ./2024 trip/IMG_1001.jpg
f318e36417590c104b60ea801214aff6fe53407e25c8439e392054e77b59d897  ./Phone backup/IMG_1001 (1).jpg
f318e36417590c104b60ea801214aff6fe53407e25c8439e392054e77b59d897  ./Phone backup/IMG_1001.jpg

This one hashes every file, so it's slower than option 1 on a big library, but it needs nothing installed.

Option 3: Czkawka (free app, also finds similar photos)

Czkawka is a free, open-source app for Windows, macOS and Linux (its newer interface is called Krokiet). Besides exact duplicates it finds similar images, similar videos and empty folders. Download it from its official GitHub page.

Option 4: TidyBatch Pro (moves copies aside, with undo)

tidybatch dedupe does the size → SHA-256 search, keeps the oldest copy of each file and moves the others into _duplicates/, keeping their folder structure so you can see where each one came from. Real output on the same test folder:

$ tidybatch dedupe ./Pictures -r
  Phone backup/IMG_1001.jpg      ->  _duplicates/Phone backup/IMG_1001.jpg
  Phone backup/IMG_1001 (1).jpg  ->  _duplicates/Phone backup/IMG_1001 (1).jpg
  report copy.pdf                ->  _duplicates/report copy.pdf

3 file(s) would change.
Dry run: nothing was changed. Add --apply to do it.

Add --apply to move them. Check _duplicates/, then delete it yourself, or put everything back with tidybatch undo ./Pictures. Nothing is ever deleted by the tool. dedupe is part of TidyBatch Pro (one-time purchase, ready-to-run apps for Windows and Linux); options 1–3 are free.

Which copy should you keep?

• The oldest (TidyBatch's default) is usually the original; later copies come from backups and re-downloads.
• The one in the "right" folder: if you have a main library, delete the copies outside it, not inside it.
• Careful with sync folders. Deleting a file inside Dropbox, OneDrive or iCloud Drive usually deletes it on every synced device too.

Which one should you use?

For similar (edited or resized) photos, use Czkawka.

Related: sort your Downloads folder by file type →

Related: can you undo a bulk rename? →