42 Open Source Data Pipeline Software Projects

A Python library that implements software engineering best-practice for data and ML pipelines.

A list of useful resources to learn Data Engineering from scratch

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Go stream processing library

A list about Apache Kafka

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

Python Data Streams

Content for architecting a data science platform for products using Luigi, Spark & Flask.

:whale: Tool to automate data quality checks on data pipelines

Data transformations for the ML era

:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript

Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.

Tools for ASR Corpus Generation from Online Video

A cross-platform tool for data pipelines.

Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena

Outbrain's data pipeline framework

一个基于浏览器环境的数据采集SDK

Model complex data transformation pipelines easily

Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline

(deprecated) A fast and memory-efficient Python data engineering framework for machine learning.

Copy Pandas DataFrames and HDF5 files to PostgreSQL database

Framework which helps you to make parallel/distributed calculations using data pipelines

OONI data processing pipeline

A Machine Learning and Real-Time Data Analytics Workshop

Network traffic data pipeline for real-time predictions and building datasets for deep neural networks

A web service for disambiguating and canonically storing entities.

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Pipeline module for parallel real-time data processing for machine learning models development and production purposes.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Blog post on ETL pipelines with Airflow

Configuration and definitions of Airflow for OpenTrials

A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable & Transformable Pipeline data processing.

Here's how to get DataQuest's Data Engineering Track missions' content to work on your localhost. Using data from my Valenbisi ARIMA modeling project, I document my steps using PostgreSQL, Postico, and the Command Line to get our DataQuest exercises running out of a Jupyter Notebook.

Automating Your Data Pipeline with Apache Airflow

Data Pipeline Toolkit for Early-Stage Startups

The open source version of the AWS Data Pipeline documentation. To provide feedback & requests for changes, submit issues in this repository, or make proposed changes & submit a pull request.

RPJiOS: RPJ's RPi OS, a sensor data platform for the Raspberry Pi built with python2.7 and redis.

Data pipeline project

Kestra is a platform to build, test, schedule & monitor complex pipelines.

Smarter data pipelines for audio.

Visualise your Kedro data pipelines.

Example end to end data engineering project.