113 Open Source Airflow Software Projects
Free and open source airflow code projects including engines, APIs, generators, and tools.
Airflow 18569 ⭐
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Incubator Dolphinscheduler 4522 ⭐
Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduling platform, dedicated to solving the complex dependencies in data processing, making the scheduling system out of the box for data processing.(分布式易扩展的可视化工作流任务调度)
Dataspherestudio 889 ⭐
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Goodreads_etl_pipeline 719 ⭐
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
San089 Udacity Data Engineering Projects 337 ⭐
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Airflow Rest API Plugin 267 ⭐
A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces
Airflow Scheduler Failover Controller 190 ⭐
A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability
Data Science Stack Cookiecutter 133 ⭐
🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)
Beyond Jupyter 133 ⭐
🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)
Airflow Spark Operator Plugin 67 ⭐
A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator
Powerdatahub Terraform Aws Airflow 61 ⭐
Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor
Docker Airflow 30 ⭐
Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/
Airflow Toolkit 32 ⭐
Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) :desktop_computer: >> [ :rocket:, :ship: ]
Incremental_training 31 ⭐
Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'
Skytrax Data Warehouse 33 ⭐
A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
Jobanalytics_and_search 18 ⭐
JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.
Data Pipelines With Apache Airflow 24 ⭐
Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
Aircan 16 ⭐
💨🥫 Load data into CKAN DataStore using Airflow as the runner. Evolution of DataPusher and Xloader.
Movalytics Data Warehouse 17 ⭐
Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
Airflow User Management Plugin 13 ⭐
A plugin for Apache Airflow that allows you to manage the users that can login
Airflow Valohai Plugin 13 ⭐
:shark: Airflow plugin to scale machine learning tasks with Valohai and get automatic version control
Terraform Ecs Fargate Airflow 12 ⭐
A Terraform template for provisioning Apache Airflow workflows on AWS ECS Fargate
Automating Your Data Pipeline With Apache Airflow 15 ⭐
Automating Your Data Pipeline with Apache Airflow
Airflow Admin Tools Plugin 10 ⭐
An Airflow Plugin that provides a new page to the standard Airflow Web Server to help you perform various operations