83 Open Source Batch Processing Software Projects

Serverless batch computing made simple.

Distributed Stream and Batch Processing

Simple API, Complex Emails (JavaMail smtp wrapper)

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

The simple, stupid batch framework for Java

BIMP. Batch Image Manipulation Plugin for GIMP.

Parallel S3 and local filesystem execution tool.

Short lived microservices with Spring Batch

Ruby's missing background and batch processing system

Simplify HPC and Batch workloads on Azure

A Free Krita plugin to batch export layers and groups with maximum flexibility. Scale, trim, export multiple copies of your layers...

*Decompile All the Things* - IDA Batch Decompile plugin and script for Hex-Ray's IDA Pro that adds the ability to batch decompile multiple files and their imports with additional annotations (xref, stack var size) to the pseudocode .c file

Automate Excel downloads in seconds. Simply does the tedious, repetitive operations for all rows of excel files step by step and reports after the job is done. It can download files from URL(s) in a column of Excel files. If a new filename is provided at column B it will rename the file before saving. It will even create sub folders if column C is filled with a valid folder name.

面向单机与分布式 OLTP/OLAP 场景的可暂停的渐进式 SQL 引擎 (只用于研究)

Asakusa Framework

Python Data Processing library

Fully automated batch processing of Revit files with your own Python or Dynamo task scripts!

Parallel SSH Tools

Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.

Multi-device OpenCL kernel load balancer and pipeliner API for C#. Uses shared-distributed memory model to keep GPUs updated fast while using same kernel on all devices(for simplicity).

Elastic data processing with Apache Pulsar and Apache Flink

Example use of AWS batch

REST API for Spring Batch using Spring Boot 2.2

A Mass Image Processing tool for Windows

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

Multi-threaded batch processing and background jobs for .NET Core

This library for Android will take any set of events and batch them up before sending it to the server. It also supports persisting the events on disk so that no event gets lost because of an app crash. Typically used for developing any in-house analytics sdk where you have to make a single api call to push events to the server but you want to optimize the calls so that the api call happens only once per x events, or say once per x minutes. It also supports exponential backoff in case of network failures

Batch-processing of Revit models using the Dynamo visual programming environment

When Apache Pulsar meets Apache Spark

Detailed documentation is available here: http://ifiscripts.readthedocs.io/en/latest/index.html

Average and morph faces online http://webmorph.org/

A simple utility for executing multiple sequential or multi-threaded applications in a single multi-node batch job

Optimal binning: monotonic binning with constraints. Support batch & stream optimal binning. Scorecard modelling and counterfactual explanations.

Windows Batch script to automate batch processing using mkvtoolnix.

A blazing fast job processing system backed by GenStage & Redis.

Tensorflow 2 Faster-RCNN implementation from scratch supporting to the batch processing with MobileNetV2 and VGG16 backbones

An applicative monad that batches commands for later more efficient execution

Unofficial alternative launcher for QuantConnect's LEAN allowing for parallel execution and looping/batching with customizable parameters and ranges.

Generic file search & replace tool, written in Python 3

Distributed, fault tolerant batch processing for Natural Language Applications and Search, using remote partitioning

Recursive Neural Networks for PyTorch

The grid-control job submission tool

Online batch converter of SVG images to Android vector drawable XML resource files

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

Youtube video and audio downloader without any installation needed

Fast asynchronous DNS batch resolver

DEPRECATED - no longer actively maintained. Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.

MarkLogic tool for processing and reporting on content, enhanced from the original CoRB

Object Oriented Modular Abstract Calibration Toolbox

”Building a concurrent and distributed system for batch processing which is fault tolerant and can scale up or scale out using Akka.NET (based on actor model)”.

Elasticsearch tool for easily collecting and batch inserting Python data and pandas DataFrames

FORMCEPT Whiteboard used by developers for developers

A command-line program for batch processing

Batch Guetzli compressions in a manageable fashion

:truck: Batch requests for Rack apps (works with Rails, Sinatra, etc)

A tiddler toolkit for Tiddlywiki 5

Run simple commands without opening AutoCAD.

Sexy ssh makes batch ssh tasks sexy!

A reliable queue for batch processing written in Go and backed by Redis.

SAPPORO is a workflow and individual task execution system. It is also useful for continuous testing of workflows.

Python scripts for the Abaqus FEA Python interpreter that allows for robust, flexible batch processing of output (.odb) files.

Linux batch scheduler supporting job memory usage limits

A tiny framework for building batch applications as a collection of tasks in a workflow.

The Proxima platform.

Sample Spring batch job for processing S3 objects one by one.

Make Flink|Spark easier!!! The original intention of StreamX is to make the development of Flink easier. StreamX focuses on the management of development phases and tasks. Our ultimate goal is to build a one-stop big data solution integrating stream processing, batch processing, data warehouse and data laker.

Extend python lists operations using .NET's LINQ syntax for clean and fast coding.

It is an efficient, flexible, and lightweight goroutine pool. It provides an easy way to deal with concurrent tasks with limited resource.

智能数据探索服务(Intelligent Data Exploration Service)

Stream processing guidelines and examples using Apache Flink and Apache Spark

Sample configuration for using Raspberry Pi's as Python Github Runners

Batch processing library for Golang.

Road to Azure Data Engineer Part-II: DP-201 - Designing an Azure Data Solution

Video Converter & Transcoder, ffmpeg frontend: 1. set options, drag multiple files, 2. transcode MKV-MP4, 3. full manual edit of ffmpeg command, 4. record network streams, 5. batch list

This is simple RPN implementation using tensorflow. It can be used for Faster-RCNN.

(Developing)FLink.CSharp - a port of Apache Flink, an open source stream processing framework with powerful stream- and batch-processing capabilities. Learn more about Flink at https://flink.apache.org.

Easily Process a Batch of Cox Models

Parallelized and efficient Node.js support for batch-mode child processes

Hands on Batch files with Spring Batch

Programming / Encode RFiD Tags with Batch - Read; Modify; Write Actions

Go bindings for Imageflow -> https://github.com/imazen/imageflow

vinyl recommendation engine based on Discogs and engineered data