121 Open Source Avro Software Projects
Free and open source avro code projects including engines, APIs, generators, and tools.
Kafka Storm Starter 729 ⭐
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Pmacct 624 ⭐
pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].
Cpp Serializers 508 ⭐
Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++
Devops Python Tools 345 ⭐
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Storagetapper 220 ⭐
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Jackson Dataformats Binary 203 ⭐
Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile
Vscode Data Preview 207 ⭐
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro & Excel data files
Bigdata Playground 163 ⭐
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Pennsignals Aptos 144 ⭐
:sunny: A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats
Gradle Avro Plugin 145 ⭐
A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.
Hpgrahsl Kafka Connect Mongodb 134 ⭐
**Unofficial / Community** Kafka Connect MongoDB Sink Connector
Avro Hadoop Starter 110 ⭐
Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.
Schemer 94 ⭐
Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
Slimmessagebus 81 ⭐
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
Open Bank Mark 76 ⭐
A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.
Avro Schema Registry 71 ⭐
Implementation of the Confluent Schema Registry API as a Rails application
Rumble 51 ⭐
⛈️ Rumble 1.8.1 "Scots pine"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
Tablasco 39 ⭐
Tablasco is a JUnit rule for comparing tables and Spark module for comparing large data sets
Kafka Connect File Pulse 47 ⭐
🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka
Orleans.streams.kafka 36 ⭐
An implementation of a PersistentStreamProvider for Microsoft Orleans and Kafka using the Confluent API
Springboot Kafka Connect Streams 33 ⭐
The main goal is to play with Kafka Connect and Streams. We have store-api that inserts/updates records in MySQL; Source connectors that monitor inserted/updated records in MySQL and push messages related to those changes to Kafka; Sink connectors that read messages from Kafka and insert documents in ES; Store-streams that listens for messages in Kafka, treats them using Kafka Streams and push new messages back to Kafka.
Fast Avro Write 31 ⭐
Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.
Avro Schema Generator 31 ⭐
Library for generating avro schema files (.avsc) based on DB tables structure
Go Kafka Avro 30 ⭐
A library provides consumer/producer to work with kafka, avro and schema registry
Avro Compatibility 29 ⭐
A user friendly API for checking for and reporting on Avro schema incompatibilities.
Amient Affinity 24 ⭐
Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka
Awesome Kafka In Clojure 25 ⭐
Awesome information for using Kafka and Confluent Platform in Clojure.
Schema Registry Php Client 24 ⭐
A PHP 7.2+ API client for the Confluent Schema Registry REST API based on Guzzle 6 - http://docs.confluent.io/current/schema-registry/docs/index.html
Daflow 22 ⭐
Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Dataclasses Avroschema 30 ⭐
Generate Avro Schemas from a Python class. Serialize and Deserialize python instances with avro schemas
Dendrites 21 ⭐
Scala library for Reactive streaming Microservices, CQRS, Event Sourcing, Event Logging, & message-driven apps.
Spring Kafka Event Sourcing Sampler 21 ⭐
Showcases how to build a small Event-sourced application using Spring Boot, Spring Kafka, Apache Avro and Apache Kafka
Confluent Avro Py 20 ⭐
An Avro SerDe implementation that integrates with the confluent schema registry and serializes and deserializes data according to the defined confluent wire format
Schema Registry Transfer Smt 22 ⭐
Kafka Connect Transform to copy Avro schemas between Schema Registries
Springboot Kafka Avro 18 ⭐
A simple Spring Boot app that talks to Kafka using Avro and Schema Registry
Ravine 18 ⭐
A bridge between synchronous HTTP workload and event-driven landscapes powered by Apache Kafka.
Kafka Scala Examples 20 ⭐
Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala
Confluent Spark Avro 17 ⭐
Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.
Springboot Kafka Debezium Ksql 19 ⭐
The goal of this project is to play with Kafka, Debezium and ksqlDB. For this, we have: research-service that inserts/updates/deletes records in MySQL; Source Connectors that monitor change of records in MySQL and push messages related to those changes to Kafka; Sink Connectors and kafka-research-consumer that listen messages from Kafka and insert/update documents in Elasticsearch; finally, ksqlDB-Server that listens to some topics in Kafka, does some joins and pushes new messages to new topics in Kafka.
Avro Serde Php 17 ⭐
Avro Serialisation/Deserialisation (SerDe) library for PHP 7.2+ with a Symfony Serializer integration
Sbt Avro 15 ⭐
Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.
Jasvorno 15 ⭐
A library for strong, schema based conversion between 'natural' JSON documents and Avro
Hello Kafka Stream Testing 16 ⭐
The most simple way to test Kafka based applications or micro-services e.g. Read/Write during HBase/Hadoop or other Data Ingestion Pipe Lines
Testing Kafka App With Jest 14 ⭐
Example set of applications using Kafka and Avro and a set of end-to-end tests in Jest
Dumping Machine 13 ⭐
Dumping Machine is an application which dumps Kafka Avro topics to S3 or HDFS as Parquet
Serialization Benchmark 12 ⭐
benchmark for modern serialization systems: Apache Avro, Protocol Buffers, Apache Thrift and MessagePack written in Scala
Camus Compressor 12 ⭐
Camus Compressor merges files created by Camus and saves them in a compressed format.
Simple Avro Kafka Golang 10 ⭐
📨 Simple Avro Kafka using Golang for Producer and Consumer with Docker
Php Kafka Lib 11 ⭐
PHP Kafka producer / consumer library with PHP Avro support, based on php-rdkafka