121 Open Source Avro Software Projects

Apache Avro is a data serialization system.

Record Query - A tool for doing record analysis and transformation

Confluent Schema Registry for Kafka

Avro for JavaScript :zap:

Apache Kafka and Confluent Platform examples and demos

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].

Avro schema generation and serialization / deserialization for Scala

Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++

Iceberg is a table format for large, slow-moving tabular data

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Web tool for Avro Schema Registry |

A tool for data sampling, data generation, and data diffing

Divolte Collector

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

Mu (μ) is a purely functional framework for building micro services.

:sunny: A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats

A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.

**Unofficial / Community** Kafka Connect MongoDB Sink Connector - Find the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

MongoDB Kafka Connector

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

Scala library to eliminate boilerplate

A library that makes it easier to use the Avro serialization format from Ruby.

Avro SerDe for Apache Spark structured APIs.

Command Line Tool for managing Apache Kafka

Lego bricks to build Apache Kafka serializers and deserializers

A CLI and Go client for Kafka Schema Registry. This repository is deprecated and no longer maintained.

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

A fast Go Avro codec

Kafka backed service library.

Kafka Connect Sink/Source for Twitter

Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.

Implementation of the Confluent Schema Registry API as a Rails application

Avro support for kotlinx.serialization

Ruby DSL to create Avro schemas

Shapeless utilities for common data types

Fluent Kafka Streams Test with Java

The metrics pipeline for DC/OS 1.9-1.11

Generate Ruby models from Avro schemas

Functional Avro for Scala

GCS support for avro-tools, parquet-tools and protobuf

⛈️ RumbleDB 1.15.0 "Ivory Palm" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

A collection of Magnolia add-on modules

A convenient Elixir library to work with Avro schemas and Confluent® Schema Registry

Tablasco is a JUnit rule for comparing tables and Spark module for comparing large data sets

A light Kafka to HDFS/S3 ETL library based on Apache Spark

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

Gradle plugin to interact with Confluent Schema-Registry.

JDK 11 examples and demo projects.

Wrangler Transform: A DMD system for transforming Big Data

An implementation of a PersistentStreamProvider for Microsoft Orleans and Kafka using the Confluent API

An avro converter for Kafka Connect without a Schema Registry

The main goal is to play with Kafka Connect and Streams. We have store-api that inserts/updates records in MySQL; Source connectors that monitor inserted/updated records in MySQL and push messages related to those changes to Kafka; Sink connectors that read messages from Kafka and insert documents in ES; Store-streams that listens for messages in Kafka, treats them using Kafka Streams and push new messages back to Kafka.

Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.

Kafka Streaming KSQL message transformation library

Goavro translates between Go native types and binary and textual Avro data

Library for generating avro schema files (.avsc) based on DB tables structure

Data Highway is a low friction, schema-on-write, data streaming platform

A library provides consumer/producer to work with kafka, avro and schema registry

A user friendly API for checking for and reporting on Avro schema incompatibilities.

:musical_score: Docker compose files for various kafka stacks

How to use Parquet in Flink

A collection of Apache Parquet add-on modules

A SQL-like language for performing ETL transformations.

An Avro implementation for .NET

Standalone alternatives to Kafka Connect Connectors

Supermodel browser library and CLI tool for working with JSON Schema data models

Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka

Use SQL to transform your avro schema/records

:orange_book: json & avro http schema registry backed by Kafka

Convert XSD -> AVSC and XML -> AVRO

Awesome information for using Kafka and Confluent Platform in Clojure.

A PHP 7.3+ API client for the Confluent Schema Registry REST API based on Guzzle 6 - http://docs.confluent.io/current/schema-registry/docs/index.html

An Avro Library that emphasizes testability and ease of use.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Generate Avro Schemas from a Python class. Serialize and Deserialize python instances with avro schemas

Infer clojure specs from any Avro type

Scala library for Reactive streaming Microservices, CQRS, Event Sourcing, Event Logging, & message-driven apps.

Showcases how to build a small Event-sourced application using Spring Boot, Spring Kafka, Apache Avro and Apache Kafka

Compile Apache Avro schema files to TypeScript classes

An Avro SerDe implementation that integrates with the confluent schema registry and serializes and deserializes data according to the defined confluent wire format

Kafka Connect Transform to copy Avro schemas between Schema Registries

Golang Client for Schema Registry

Generate json-schemas for flowtype definitions

A simple Spring Boot app that talks to Kafka using Avro and Schema Registry

A bridge between synchronous HTTP workload and event-driven landscapes powered by Apache Kafka.

:sunny: Avro, Protobuf, Thrift on Swagger

Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala

Avrocado is a convenience library to handle Avro in Golang

Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.

The goal of this project is to play with Kafka, Debezium and ksqlDB. For this, we have: research-service that inserts/updates/deletes records in MySQL; Source Connectors that monitor change of records in MySQL and push messages related to those changes to Kafka; Sink Connectors and kafka-research-consumer that listen messages from Kafka and insert/update documents in Elasticsearch; finally, ksqlDB-Server that listens to some topics in Kafka, does some joins and pushes new messages to new topics in Kafka.

Kafka De/Serializer using avro and Confluent's Schema Registry

Avro Schema Evolution made easy

Avro Serialisation/Deserialisation (SerDe) library for PHP 7.3+ & 8.0 with a Symfony Serializer integration

Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.

Visualizer for Avro Schemas (.avsc) - Try it yourself at: