Signalilo is our Alertmanager to Icinga2 bridge implementation. Signalilo acts on webhooks which it receives from Alertmanager and forwards the alerts in those webhooks to Icinga2 using https://github.com/vshn/go-icinga2-client.
See CHANGELOG.md for changelogs of each release version of Signalilo.
See DockerHub for pre-built Docker images of Signalilo
Signalilo gets started from the command line and takes its configuration
either as options or as environment variables. Use
signalilo --help to get a
list of all available configuration parameters.
When started, Signalilo listens to HTTP requests on the following paths:
/webhookEndpoint to accept alerts from Alertmanager.
/healthzreturns HTTP 200 with
okas its payload as long as the webhook serving loop is operational.
helm install --name signalilo appuio/signalilo
docker run --name signalilo vshn/signalilo
The Helm chart should work on OpenShift
SIGNALILO_UUID: UUID which identifies the Signalilo instance.
SIGNALILO_ICINGA_HOSTNAME: Name of the Servicehost in Icinga2.
SIGNALILO_ICINGA_URL: URL of the Icinga API.
SIGNALILO_ICINGA_USERNAME: Authentication against Icinga2 API.
SIGNALILO_ICINGA_PASSWORD: Authentication against Icinga2 API.
SIGNALILO_LOG_LEVEL: Integer to control verbosity of logging (default: 2).
SIGNALILO_ICINGA_INSECURE_TLS: If true, disable strict TLS checking of Icinga2 API SSL certificate (default: false).
SIGNALILO_ICINGA_DISABLE_KEEPALIVES: If true, disable http keep-alives with Icinga2 API and will only use the connection to the server for a single HTTP request (default: false).
SIGNALILO_ICINGA_DEBUG: If true, enable debugging mode in Icinga client (default: false).
SIGNALILO_ICINGA_GC_INTERVAL: Interval to run Garbage collection of recovered alerts in Icinga (default 15m).
SIGNALILO_ICINGA_HEARTBEAT_INTERVAL: Interval to send heartbeat to Icinga (default 60s).
SIGNALILO_ICINGA_KEEP_FOR: How long to keep Icinga2 services around after they transition to state OK (default 168h).
SIGNALILO_ICINGA_CA: A PEM string of the trusted CA certificate for the Icinga2 API certificate.
SIGNALILO_ALERTMANAGER_PORT: Port on which Signalilo listens to incoming webhooks (default 8888).
SIGNALILO_ALERTMANAGER_BEARER_TOKEN: Incoming webhook authentication. Can be either set via
Authorizationheader or in the
tokenURL query parameter.
SIGNALILO_ALERTMANAGER_TLS_CERT: Path of certificate file for TLS-enabled webhook endpoint. Should contain the full chain.
SIGNALILO_ALERTMANAGER_TLS_KEY: Path of private key file for TLS-enabled webhook endpoint. TLS is enabled when both TLS_CERT and TLS_KEY are set.
Integration to Prometheus/Alertmanager.
/webhook accepts alerts in the format of Alertmanager.
The following Alertmanager configuration is an example taken from a Signalilo
installation on OpenShift.
global: resolve_timeout: 5m route: group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: default routes: - match: alertname: DeadMansSwitch repeat_interval: 5m receiver: deadmansswitch receivers: - name: default webhook_configs: - send_resolved: true http_config: bearer_token: "*****" url: http://signalilo.appuio-monitoring/webhook - name: deadmansswitch
Signalilo requires a set of information to be part of an alert. Without this information, the check generated in Icinga will be lacking.
severity: Must be one of
description: mapped to
message: mapped to
Integration with Icinga
Service objects in Icinga will get garbage collected (aka deleted) on a regular basis, following these rules:
- Service object is in OK state
- Last transition to OK state was more than "keep_for" ago
- UUID of app matches "vars.bridge_uuid"
All state needed for doing garbage collection is stored in Icinga service variables.
On startup, Signalilo checks if the matching heartbeat service is available in Icinga, otherwise it exits with a fatal error. During operation, Signalilo regularly posts its state to the heartbeat service. If no state update was provided, Icinga automatically marks the check as UNKNOWN. See Icinga2 passive checks for a description of how the service object needs to be configured.
All labels and annotations will be mapped to custom variables. Keys of Labels
will be prefixed with
label_ and keys of annotations with
If the key an annotation or label starts with
icinga_ it will also be added
as custom variable but without any prefix. Since all labels and annotations
will be strings, a type information needs to be provided so that a conversion
can be done accordingly. This is done by adding the type as part of the prefix
icinga_<type>_). Current supported types are
icinga_foo_string-> label/annotation named
foowith value is passed as is.
icinga_bar_number-> label/annotation named
barwith its value is converted to an integer number.
In case there is a label and an annotation with the
icinga_<type> prefix, the
value of the annotation will take precedence in the resulting set of custom
Signalilo supports creating heartbeat services in Icinga. This can be used to
map alerts like the
DeadMansSwitch which comes with
and signals that the whole Prometheus stack is healthy.
In order for Signalilo to treat an alert as a heartbeat, the alert must have
heartbeat. Signalilo will try to parse the value of that label as a
If the value is parsed successfully, Signalilo will create a Icinga service check with active checks enabled and with the check interval set to the the parsed duration plus ten percent. We add ten percent to the parsed duration to account for network latencies etc, which could otherwise lead to flapping heartbeat checks.