74 Open Source Deduplication Software Projects

Fast, secure, efficient backup program

Deduplicating archiver with compression and authenticated encryption.

Prometheus Alertmanager

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

Find duplicate files

Extremely fast tool to remove duplicates and other lint from your filesystem

Simple, configuration-driven backup software for servers and workstations

Data deduplication engine, supporting optional compression and public key encryption.

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

A toolkit for record linkage and duplicate detection in Python

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

A list of free data matching and record linkage software.

A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

Quickly detect already witnessed data.

Userspace tools for managing VDO volumes.

Spark RDD with Lucene's query and entity linkage capabilities

Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.

You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash

Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.

Record Linkage ToolKit (Find and link entities)

CLI utility to find duplicate files

Dedupe/batch geocode addresses and venues around the world with libpostal

The Dropbox for IPFS (without the icky stuff)

Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices

Resources for tackling record linkage / deduplication / data matching problems

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters

A simple command line interface to the datamade/dedupe library.

FastCDC implementation in Rust

Self-contained C# library for data deduplication using Sqlite

Continuous data protection for GNU/Linux (cdpfgl).

A backup program that does deduplication, compression, encryption

A Scalable Data Cleaning Library for PySpark.

CLI tool for image duplicate detection

Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.

:bar_chart: python package for performing deduplication using flexible text matching and cleaning in pandas dataframe

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

CLI utility to find near duplicate images and remove all but the best copy.

Spark Search - high performance advanced search features based on Apache Lucene

Deduplicating filesystem via Python3, FUSE and SQLite

A browser user interface for manual labeling of record pairs.

Fast block-level out-of-band BTRFS deduplication tool.

New project: https://git.sr.ht/~tsileo/blobfs

An efficient backup tool inspired by Git, saving your bandwidth and providing global deduplication at file level.

Fork of the Freely Extensible Biomedical Record Linkage program

A command-line tool for deduplicating entries in a file or stream.

A library for secure, deduplicated, transactional, and verifiable data storage

A Qt frontend for the command line software BorgBackup.

📧 CLI to deduplicate mails from mail boxes.

Cargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.

Config driven, easy backup cli for restic.

Scalable data mastering, deduplication and entity resolution.

PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.

Generate duplex/single consensus reads to reduce sequencing noises and remove duplications

Automation for the serious data hoarder that wants to have their data and use it

Yet Another Dupes Finder

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

FastCDC implementation in Python https://pypi.org/project/fastcdc/

Java DSL for (online) deduplication

A maximum-strength name parser for record linkage.

Content-Addressable File System (used by BitWrk)

图片查重、图片去重、Find/Delete duplicated images

Backend (Docker & API) for matchID project

Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.

Perceptual hashing command-line tool

Webpack plugin that resolves copies of the same module in different locations to a single path.

A fast high compression read-only file system

Efficiently import pictures while handling duplicates gracefully

A Management Friendly Deduplication Prototype System for Backup

Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, обслуживание БД)

S3 backed FUSE Filesystem written in Go with dedup and encryption.

Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.