Open Source Libraries 👉Dedupe

21 Open Source Dedupe Software Projects

Free and open source dedupe code projects including engines, APIs, generators, and tools.

Restic 15513 ⭐
Fast, secure, efficient backup program
Borg 7912 ⭐
Deduplicating archiver with compression and authenticated encryption.
Dedupe 3268 ⭐
:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
Yarn Deduplicate 1107 ⭐
Deduplication tool for yarn.lock files
Free Style 697 ⭐
Make CSS easier and more maintainable by using JavaScript
Jdupes 1011 ⭐
A powerful duplicate file finder and an enhanced fork of 'fdupes'.
Recordlinkage 625 ⭐
A toolkit for record linkage and duplicate detection in Python
Csvdedupe 342 ⭐
:id: Command line tool for deduplicating CSV files
Dedupe Examples 306 ⭐
:id: Examples for using the dedupe library
Dupe Krill 123 ⭐
A fast file deduplicator
Maildir Deduplicate 121 ⭐
📧 CLI to deduplicate mails from mail boxes.
Imgdupes 213 ⭐
Finding and deleting near-duplicate images based on perceptual hash.
Pgdedupe 39 ⭐
A simple command line interface to the datamade/dedupe library.
Watsondedupe 31 ⭐
Self-contained C# library for data deduplication using Sqlite
Django Super Deduper 25 ⭐
Utilities for de-duping Django model instances
Duplex 19 ⭐
Duplicate code finder for Elixir
Dduper 100 ⭐
Fast block-level out-of-band BTRFS deduplication tool.
Duplicut 515 ⭐
Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)
Zingg 371 ⭐
Scalable data mastering, deduplication and entity resolution.
Zygo Bees 268 ⭐
Best-Effort Extent-Same, a btrfs dedupe agent
Yadf 26 ⭐
Yet Another Dupes Finder