314 Open Source Etl Software Projects

Fancy stream processing made operationally mundane

Linq to database provider.

A Python stream processing engine modeled after Yahoo! Pipes

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Data processing & ETL framework for Ruby

Sync data between persistence engines, like ETL only not stodgy

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Actively curated list of awesome BI tools. PRs welcome!

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

[DEPRECATED] Detect threats with log data and improve cloud security posture

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

React components to build CSV files on the fly basing on Array/literal object of data

This repository is a getting started guide to Singer.

Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.

A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.

Example project implementing best practices for PySpark ETL jobs and applications.

:crystal_ball: Transform, query, and download geospatial data on the web.

A serverless cluster computing system for the Go programming language

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

A lightweight stream processing library for Go

The premier open source Data Quality solution

Power of appbase.io via CLI, with nifty imports from your favorite data sources

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

WeDataSphere is a financial grade, one-stop big data platform suite.

Extensible data integration Java framework for building XML and non-XML fragment-based applications

A simplified, lightweight ETL Framework based on Apache Spark

ETL Library for Machine Learning - data pipelines, data munging and wrangling

基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具

:blue_book: Making Data, the DataMade Way

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

R client for the Elasticsearch HTTP API

Example DAGs using hooks and operators from Airflow Plugins

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

A MongoDB to Elasticsearch connector

A cross-platform command line tool for parallelised content extraction and analysis.

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

Categorical Query Language IDE

Linked Data & RDF Manufacturing Tools in Clojure

Bender - Serverless ETL Framework

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

An example mini data warehouse for python project stats, template for new projects

mito ETL tool

数据可视化, 数据挖掘, 数据处理 ETL

A visual ETL development and debugging tool for big data

Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & Named Entity Recognition) & data enrichment (annotation) pipelines & ingestor to Solr or Elastic search index & linked data graph database

Big Data Toolkit for the JVM

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure :dizzy: 多语言执行器，分布式爬虫

The MarkLogic Data Hub: documentation ==>

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Configurable Extract, Transform, and Load

Mass processing data with a complete ETL for .net developers

Python ETL(Extract-Transform-Load) tool / Data migration tool

基于spring boot通过java代码调用kette

equivalent to kafka-connect :wrench: for nodejs :sparkles::turtle::rocket::sparkles:

Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer） 分布式运行（DataX on YARN） 功能

The CSV to database command line loader

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile your test suite. Just create an Xml file and let the framework interpret it and play your tests. The framework is designed as an add-on of NUnit but with the possibility to port it easily to other testing frameworks.

LinkedPipes ETL is an RDF based, lightweight ETL tool

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

Česká otevřená data

A luigi powered analytics / warehouse stack

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

A tool for building feature stores.

Thain is a distributed flow schedule platform.

Ohara - Easy to deploy the streaming application

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Udacity Data Engineering Nano Degree (DEND)

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

locopy: Loading/Unloading to Redshift and Snowflake using Python.

ETL with Python - Taught at DWH course 2017 (TAU)

Nextract is a Extract Transform Load (ETL) platform build on top of Node.js streams

A Singer.io Target for Postgres

A Django app to download, extract and load campaign finance and lobbying activity data from the California Secretary of State's CAL-ACCESS database

ETLy is an add-on dashboard service on top of Apache Airflow.

LSC engine

A simple Spark-powered ETL framework that just works 🍺

Tool and library to convert ETW logs to JSON files

an open-source data management platform for knowledge workers (https://github.com/dswarm/dswarm-documentation/wiki)

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

Highly performant JavaScript data stream ETL engine.

Lightweight library to write, orchestrate and test your SQL ETL. Writing ETL with data integrity in mind.

An Elasticsearch client tailored to data science workflows.

Open Source ETL designed for and dedicated to Log processing and transformation

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

Business Intelligence and Data Warehousing

Ruby scripting for pentaho-kettle

PHP ETL (Extract / Transform / Load) library with SOLID principles + almost no dependency.

an data-centric integration platform

A light Kafka to HDFS/S3 ETL library based on Apache Spark

Yet Another ETL in PHP

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

code, labs and lectures for the course