52 Open Source Deduplication Software Projects

Fast, secure, efficient backup program

Deduplicating archiver with compression and authenticated encryption.

Prometheus Alertmanager

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

Find duplicate files

Extremely fast tool to remove duplicates and other lint from your filesystem

Simple, configuration-driven backup software for servers and workstations

Data deduplication engine, supporting optional compression and public key encryption.

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

A toolkit for record linkage and duplicate detection in Python

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

A list of free data matching and record linkage software.

A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

Quickly detect already witnessed data.

Userspace tools for managing VDO volumes.

Spark RDD with Lucene's query and entity linkage capabilities

Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.

You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash

Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.

Record Linkage ToolKit (Find and link entities)

CLI utility to find duplicate files

Dedupe/batch geocode addresses and venues around the world with libpostal

The Dropbox for IPFS (without the icky stuff)

Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices

Resources for tackling record linkage / deduplication / data matching problems

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

Implementation in Apache Spark of the EM algorithm to estimate parameters of Fellegi-Sunter's canonical model of record linkage.

A simple command line interface to the datamade/dedupe library.

FastCDC implementation in Rust

Self-contained C# library for data deduplication using Sqlite

Continuous data protection for GNU/Linux (cdpfgl).

A backup program that does deduplication, compression, encryption

A Scalable Data Cleaning Library for PySpark.

CLI tool for image duplicate detection

Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI).

:bar_chart: python package for performing deduplication using flexible text matching and cleaning in pandas dataframe

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

CLI utility to find near duplicate images and remove all but the best copy.

Spark Search - high performance advanced search features based on Apache Lucene

Deduplicating filesystem via Python3, FUSE and SQLite

A browser user interface for manual labeling of record pairs.

Fast block-level out-of-band BTRFS deduplication tool.

New project: https://git.sr.ht/~tsileo/blobfs

An efficient backup tool inspired by Git, saving your bandwidth and providing global deduplication at file level.

Fork of the Freely Extensible Biomedical Record Linkage program

A command-line tool for deduplicating entries in a file or stream.

A library for secure, deduplicated, transactional, and verifiable data storage

A Qt frontend for the command line software BorgBackup.

:e-mail: CLI to deduplicate mails from maildir folders

Cargo with less noise: warnings are skipped until errors are fixed, etc.