72 Open Source Data Pipeline Software Projects

A Python framework for creating reproducible, maintainable and modular data science code.

A list of useful resources to learn Data Engineering from scratch

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

A lightweight stream processing library for Go

A list about Apache Kafka

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

Streaming reactive and dataflow graphs in Python

Content for architecting a data science platform for products using Luigi, Spark & Flask.

:whale: Tool to automate data quality checks on data pipelines

Data transformations for the ML era

:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Tools for ASR Corpus Generation from Online Video

A cross-platform tool for data pipelines.

Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena

Outbrain's data pipeline framework

一个基于浏览器环境的数据采集SDK

Model complex data transformation pipelines easily

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

(deprecated) A fast and memory-efficient Python data engineering framework for machine learning.

Copy Pandas DataFrames and HDF5 files to PostgreSQL database

Framework which helps you to make parallel/distributed calculations using data pipelines

OONI data processing pipeline

A Machine Learning and Real-Time Data Analytics Workshop

Network traffic data pipeline for real-time predictions and building datasets for deep neural networks

A web service for disambiguating and canonically storing entities.

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Blog post on ETL pipelines with Airflow

Configuration and definitions of Airflow for OpenTrials

A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable & Transformable Pipeline data processing.

Here's how to get DataQuest's Data Engineering Track missions' content to work on your localhost. Using data from my Valenbisi ARIMA modeling project, I document my steps using PostgreSQL, Postico, and the Command Line to get our DataQuest exercises running out of a Jupyter Notebook.

Automating Your Data Pipeline with Apache Airflow

Data Pipeline Toolkit for Early-Stage Startups

The open source version of the AWS Data Pipeline documentation. To provide feedback & requests for changes, submit issues in this repository, or make proposed changes & submit a pull request.

RPJiOS: RPJ's RPi OS, a sensor data platform for the Raspberry Pi built with python2.7 and redis.

Data pipeline project

Kestra, the modern, scalable orchestrator & scheduler open source platform.

Smarter data pipelines for audio.

Visualise your Kedro data pipelines.

Example end to end data engineering project.

The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP

Profile and monitor your ML data pipeline end-to-end , Join us in slack @ http://join.slack.whylabs.ai/

Use SQL to build ELT pipelines on a data lakehouse.

Fluent data pipelines for python and your shell

Elementary is an open-source data observability framework for modern data teams, starting with data lineage.

Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management

:fire: Data pipeline and automation tool.

Making DAG construction easier

Build and deploy a serverless data pipeline on AWS with no effort.

Data pipelines from re-usable components

A Clojure machine learning library

(project & tutorial) dag pipeline tests + ci/cd setup

Automated Tool for Optimized Modelling

Focus on Algorithm Design, Not on Data Wrangling

Real estate dagster pipeline

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.

Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes

Distributed DataLoader For Pytorch Based On Ray

A fully-featured multi-source data pipeline for continuously extracting knowledge from COVID-19 data.

This project describes how to write full ETL data pipeline using spark.

Rivery CLI

A GUI frontend to manage blockchain ingestion with slurp

This repository holds the python files and notebooks associated with the Udacity Data Engineering Nanodegree.

A real-time news scraping and recommendation system

The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย

Ordered-concurrently a library for parallel processing with ordered output in Go. Process work concurrently / in parallel and returns output in a channel in the order of input. It is useful in concurrently / parallelly processing items in a queue, and get output in the order provided by the queue.

A fully incremental model, that transforms raw web event data generated by the Snowplow JavaScript tracker into a series of derived tables of varying levels of aggregation.

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

task management & automation tool