154 Open Source Avro Software Projects
Free and open source avro code projects including engines, APIs, generators, and tools.
Kafka Storm Starter 725 ⭐
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Pmacct 786 ⭐
pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].
Cpp Serializers 597 ⭐
Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++
Devops Python Tools 521 ⭐
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Storagetapper 271 ⭐
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Jackson Dataformats Binary 256 ⭐
Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile
Vscode Data Preview 362 ⭐
Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files
Bigdata Playground 192 ⭐
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Pennsignals Aptos 143 ⭐
:sunny: A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats
Gradle Avro Plugin 214 ⭐
A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.
Hpgrahsl Kafka Connect Mongodb 143 ⭐
**Unofficial / Community** Kafka Connect MongoDB Sink Connector - Find the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Avro Hadoop Starter 112 ⭐
Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.
Lensesio Schema Registry 111 ⭐
A CLI and Go client for Kafka Schema Registry. This repository is deprecated and no longer maintained.
Schemer 108 ⭐
Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
Slimmessagebus 210 ⭐
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
Open Bank Mark 87 ⭐
A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.
Avro Schema Registry 77 ⭐
Implementation of the Confluent Schema Registry API as a Rails application
Rumble 93 ⭐
⛈️ RumbleDB 1.16.0 "Shagbark Hickory" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
Tablasco 45 ⭐
Tablasco is a JUnit rule for comparing tables and Spark module for comparing large data sets
Kafka Connect File Pulse 174 ⭐
🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka
Orleans.streams.kafka 49 ⭐
An implementation of a PersistentStreamProvider for Microsoft Orleans and Kafka using the Confluent API
Springboot Kafka Connect Streams 61 ⭐
The main goal is to play with Kafka Connect and Streams. We have store-api that inserts/updates records in MySQL; Source connectors that monitor inserted/updated records in MySQL and push messages related to those changes to Kafka; Sink connectors that read messages from Kafka and insert documents in ES; Store-streams that listens for messages in Kafka, treats them using Kafka Streams and push new messages back to Kafka.
Fast Avro Write 32 ⭐
Writing an Avro file is not as fast as you might want it. This is a library to write considerably faster to an avro file.
Avro Schema Generator 38 ⭐
Library for generating avro schema files (.avsc) based on DB tables structure
Go Kafka Avro 46 ⭐
A library provides consumer/producer to work with kafka, avro and schema registry
Avro Compatibility 41 ⭐
A user friendly API for checking for and reporting on Avro schema incompatibilities.
Amient Affinity 26 ⭐
Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka
Awesome Kafka In Clojure 26 ⭐
Awesome information for using Kafka and Confluent Platform in Clojure.
Schema Registry Php Client 35 ⭐
A PHP 7.3+ API client for the Confluent Schema Registry REST API based on Guzzle 6 - http://docs.confluent.io/current/schema-registry/docs/index.html
Daflow 26 ⭐
Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Dataclasses Avroschema 89 ⭐
Generate Avro Schemas from a Python class. Serialize and Deserialize python instances with avro schemas
Dendrites 21 ⭐
Scala library for Reactive streaming Microservices, CQRS, Event Sourcing, Event Logging, & message-driven apps.
Spring Kafka Event Sourcing Sampler 32 ⭐
Showcases how to build a small Event-sourced application using Spring Boot, Spring Kafka, Apache Avro and Apache Kafka
Confluent Avro Py 22 ⭐
An Avro SerDe implementation that integrates with the confluent schema registry and serializes and deserializes data according to the defined confluent wire format
Schema Registry Transfer Smt 34 ⭐
Kafka Connect Transform to copy Avro schemas between Schema Registries
Springboot Kafka Avro 30 ⭐
A simple Spring Boot app that talks to Kafka using Avro and Schema Registry
Ravine 18 ⭐
A bridge between synchronous HTTP workload and event-driven landscapes powered by Apache Kafka.
Kafka Scala Examples 37 ⭐
Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala
Confluent Spark Avro 18 ⭐
Spark UDFs to deserialize Avro messages with schemas stored in Schema Registry.
Springboot Kafka Debezium Ksql 39 ⭐
The goal of this project is to play with Kafka, Debezium and ksqlDB. For this, we have: research-service that inserts/updates/deletes records in MySQL; Source Connectors that monitor change of records in MySQL and push messages related to those changes to Kafka; Sink Connectors and kafka-research-consumer that listen messages from Kafka and insert/update documents in Elasticsearch; finally, ksqlDB-Server that listens to some topics in Kafka, does some joins and pushes new messages to new topics in Kafka.
Avro Serde Php 33 ⭐
Avro Serialisation/Deserialisation (SerDe) library for PHP 7.3+ & 8.0 with a Symfony Serializer integration
Sbt Avro 15 ⭐
Plugin SBT to Generate Scala classes from Apache Avro schemas hosted on a remote Confluent Schema Registry.