154 Open Source Avro Software Projects

Kafka De/Serializer using avro and Confluent's Schema Registry

⛈️ RumbleDB 1.16.0 "Shagbark Hickory" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.

An Avro Library that emphasizes testability and ease of use.

Showcases how to build a small Event-sourced application using Spring Boot, Spring Kafka, Apache Avro and Apache Kafka

Generate json-schemas for flowtype definitions

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

Iceberg is a table format for large, slow-moving tabular data

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

A library provides consumer/producer to work with kafka, avro and schema registry

Kafka backed service library.

Avrocado is a convenience library to handle Avro in Golang

Compile Apache Avro schema files to TypeScript classes

An avro converter for Kafka Connect without a Schema Registry

An implementation of a PersistentStreamProvider for Microsoft Orleans and Kafka using the Confluent API

Apache Kafka and Confluent Platform examples and demos

Avro SerDe for Apache Spark structured APIs.

Gradle plugin to interact with Confluent Schema-Registry.

Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++

A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.

JDK 11 examples and demo projects.

Awesome information for using Kafka and Confluent Platform in Clojure.

Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala

Avro Serialisation/Deserialisation (SerDe) library for PHP 7.3+ & 8.0 with a Symfony Serializer integration

Avro Schema Evolution made easy

Command Line Tool for managing Apache Kafka

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

Fluent Kafka Streams Test with Java

The main goal is to play with Kafka Connect and Streams. We have store-api that inserts/updates records in MySQL; Source connectors that monitor inserted/updated records in MySQL and push messages related to those changes to Kafka; Sink connectors that read messages from Kafka and insert documents in ES; Store-streams that listens for messages in Kafka, treats them using Kafka Streams and push new messages back to Kafka.

Supermodel browser library and CLI tool for working with JSON Schema data models

The goal of this project is to play with Kafka, Debezium and ksqlDB. For this, we have: research-service that inserts/updates/deletes records in MySQL; Source Connectors that monitor change of records in MySQL and push messages related to those changes to Kafka; Sink Connectors and kafka-research-consumer that listen messages from Kafka and insert/update documents in Elasticsearch; finally, ksqlDB-Server that listens to some topics in Kafka, does some joins and pushes new messages to new topics in Kafka.

Kafka Connect Transform to copy Avro schemas between Schema Registries

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

Recursively infer clojure spec from any (nested) org.apache.avro.Schema

A fast Go Avro codec

MongoDB Kafka Connector

A bridge between synchronous HTTP workload and event-driven landscapes powered by Apache Kafka.

A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.

Functional Avro for Scala

Visualizer for Avro Schemas (.avsc) - Try it yourself at:

Avro support for kotlinx.serialization

A convenient Elixir library to work with Avro schemas and Confluent® Schema Registry

Golang Client for Schema Registry

A simple Spring Boot app that talks to Kafka using Avro and Schema Registry

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

An Avro implementation for .NET

Generate Avro Schemas from a Python class. Serialize and Deserialize python instances with avro schemas

Apache Avro is a data serialization system.

Mu (μ) is a purely functional framework for building micro services.

A collection of Magnolia add-on modules

Divolte Collector

Standalone alternatives to Kafka Connect Connectors

An Avro SerDe implementation that integrates with the confluent schema registry and serializes and deserializes data according to the defined confluent wire format

A collection of Apache Parquet add-on modules

Scala library for Reactive streaming Microservices, CQRS, Event Sourcing, Event Logging, & message-driven apps.

Confluent Schema Registry for Kafka

How to use Parquet in Flink

A library that makes it easier to use the Avro serialization format from Ruby.

Avro for JavaScript :zap:

Avro schema generation and serialization / deserialization for Scala

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].

Record Query - A tool for doing record analysis and transformation

Kafka Connect Sink/Source for Twitter

Ruby DSL to create Avro schemas

Implementation of the Confluent Schema Registry API as a Rails application

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

Generate Ruby models from Avro schemas

Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

Web tool for Avro Schema Registry |

A CLI and Go client for Kafka Schema Registry. This repository is deprecated and no longer maintained.

A tool for data sampling, data generation, and data diffing

Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka

GCS support for avro-tools, parquet-tools and protobuf

The metrics pipeline for DC/OS 1.9-1.11

A light Kafka to HDFS/S3 ETL library based on Apache Spark

Shapeless utilities for common data types

Wrangler Transform: A DMD system for transforming Big Data

Scala library to eliminate boilerplate

Lego bricks to build Apache Kafka serializers and deserializers

:musical_score: Docker compose files for various kafka stacks

Convert XSD -> AVSC and XML -> AVRO

**Unofficial / Community** Kafka Connect MongoDB Sink Connector - Find the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

Library for generating avro schema files (.avsc) based on DB tables structure

Kafka Streaming KSQL message transformation library

Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.

Goavro translates between Go native types and binary and textual Avro data

:sunny: Avro, Protobuf, Thrift on Swagger

A SQL-like language for performing ETL transformations.

Use SQL to transform your avro schema/records

:orange_book: json & avro http schema registry backed by Kafka

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

A PHP 7.3+ API client for the Confluent Schema Registry REST API based on Guzzle 6 - http://docs.confluent.io/current/schema-registry/docs/index.html

Tablasco is a JUnit rule for comparing tables and Spark module for comparing large data sets

Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.

A user friendly API for checking for and reporting on Avro schema incompatibilities.

:sunny: A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats