66 Open Source Batch Processing Software Projects

Serverless batch computing made simple.

Distributed Stream and Batch Processing

Simple API, Complex Emails (JavaMail smtp wrapper)

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

The simple, stupid batch framework for Java

BIMP. Batch Image Manipulation Plugin for GIMP.

Parallel S3 and local filesystem execution tool.

Short lived microservices with Spring Batch

Ruby's missing background and batch processing system

Simplify HPC and Batch workloads on Azure

A Free Krita plugin to batch export layers and groups with maximum flexibility. Scale, trim, export multiple copies of your layers...

*Decompile All the Things* - IDA Batch Decompile plugin and script for Hex-Ray's IDA Pro that adds the ability to batch decompile multiple files and their imports with additional annotations (xref, stack var size) to the pseudocode .c file

Simply does the tedious, repetitive operations for all rows of an excel file step by step and reports after the job is done. For example it can download URL(s) in a column of an Excel file. If a new filename is provided at column B it will rename the file before saving.

面向OLTP、OLAP、批处理、流处理场景的大一统SQL引擎

Asakusa Framework

Python Data Processing library

Fully automated batch processing of Revit files with your own Python or Dynamo task scripts!

Parallel SSH Tools

Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.

Multi-device OpenCL kernel load balancer and pipeliner API for C#. Uses shared-distributed memory model to keep GPUs updated fast while using same kernel on all devices(for simplicity).

Elastic data processing with Apache Pulsar and Apache Flink

Example use of AWS batch

REST API for Spring Batch using Spring Boot 2.2

A Mass Image Processing tool for Windows

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

Multi-threaded batch processing and background jobs for .NET Core

This library for Android will take any set of events and batch them up before sending it to the server. It also supports persisting the events on disk so that no event gets lost because of an app crash. Typically used for developing any in-house analytics sdk where you have to make a single api call to push events to the server but you want to optimize the calls so that the api call happens only once per x events, or say once per x minutes. It also supports exponential backoff in case of network failures

Batch-processing of Revit models using the Dynamo visual programming environment

When Apache Pulsar meets Apache Spark

Detailed documentation is available here: http://ifiscripts.readthedocs.io/en/latest/index.html

Average and morph faces online

A simple utility for executing multiple sequential or multi-threaded applications in a single multi-node batch job

Optimal binning: monotonic binning with constraints. Support batch & stream optimal binning

Windows Batch script to automate batch processing using mkvtoolnix.

A blazing fast job processing system backed by GenStage & Redis.

Tensorflow 2 Faster-RCNN implementation from scratch supporting to the batch processing with MobileNetV2 and VGG16 backbones

An applicative monad that batches commands for later more efficient execution

Unofficial alternative launcher for QuantConnect's LEAN allowing for parallel execution and looping/batching with customizable parameters and ranges.

Generic file search & replace tool, written in Python 3

Distributed, fault tolerant batch processing for Natural Language Applications and Search, using remote partitioning

Recursive Neural Networks for PyTorch

The grid-control job submission tool

Online batch converter of SVG images to Android vector drawable XML resource files

Road to Azure Data Engineer Part-I: DP-200 - Implementing an Azure Data Solution

Youtube video and audio downloader without any installation needed

Fast asynchronous DNS batch resolver

DEPRECATED - no longer actively maintained. Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.

MarkLogic tool for processing and reporting on content, enhanced from the original CoRB

Object Oriented Modular Abstract Calibration Toolbox

”Building a concurrent and distributed system for batch processing which is fault tolerant and can scale up or scale out using Akka.NET (based on actor model)”.

Elasticsearch buffer for collecting and batch inserting Python data and pandas DataFrames

FORMCEPT Whiteboard used by developers for developers

A command-line program for batch processing

Batch Guetzli compressions in a manageable fashion

:truck: Batch requests for Rack apps (works with Rails, Sinatra, etc)

A tiddler toolkit for Tiddlywiki 5

Run simple commands without opening AutoCAD.

Sexy ssh makes batch ssh tasks sexy!

A reliable queue for batch processing written in Go and backed by Redis.

SAPPORO is a workflow and individual task execution system. It is also useful for continuous testing of workflows.

Python scripts for the Abaqus FEA Python interpreter that allows for robust, flexible batch processing of output (.odb) files.

Linux batch scheduler supporting job memory usage limits

A tiny framework for building batch applications as a collection of tasks in a workflow.

The Proxima platform.

Sample Spring batch job for processing S3 objects one by one.