Amazon ECS shim loggers for containerd

Amazon ECS shim loggers for containerd is a collection of containerd compatible logger implementations that send container logs to various destinations. The following destinations are currently supported:

Build

Make sure you have golang installed. Then simply run make build to build the respective binaries. You might need to execute make get-deps to install some of the dependencies.

Usage

Containerd supports shim plugins that redirect container output to a custom binary on Linux using STDIO URIs with runc v2 runtime. These loggers can be used either programmatically or with the ctr tool.

When using the NewTask API

When using the NewTask API to start a container, simply provide the path to the built binary file shim-loggers-for-containerd and required arguments. Note it's a good practice to clean up container resources with Delete API call after container exited as the container IO pipes are not closed if the shim process is still running.

Example:

NewTask(context, cio.BinaryIO("/usr/bin/shim-loggers-for-containerd", args))

When using the ctr tool

When using ctr tool to run a container, provide the URI path to the binary file shim-loggers-for-containerd and required arguments as part of the path.

Example:

ctr run \
    --runtime io.containerd.runc.v2 \
    --log-uri "binary:///usr/bin/shim-loggers-for-containerd?--log-driver=awslogs&--arg1=value1&-args2=value2" \
    docker.io/library/redis:alpine \
    redis

Arguments

Common arguments

The following list of arguments apply to all of the shim logger binaries in this repo:

Name Required Description
log-driver Yes The name of the shim logger. Can be any of awslogs, splunk or fluentd.
container-id Yes The container id
container-name Yes The name of the container
mode No Either blocking or non-blocking. In the non-blocking mode, log events are buffered and the application continues to execute even if these logs can't be drained or sent to the destination. Logs could also be lost when the buffer is full.
max-buffer-size No Only supported in non-blocking mode. Set to 1m (1MiB) by default. Example values: 200, 4k, 1m etc.
uid No Set a custom uid for the shim logger process. 0 is not supported.
gid No Set a custom gid for the shim logger process. 0 is not supported.
cleanup-time No Set a custom time for the shim logger process clean up itself. Set to 5s (5 seconds) by default. Note the maximum supported value is 12 seconds, since containerd shim sets shim logger cleanup timeout value as 12 seconds. See reference.
container-image-id No The container image id. This is part of the docker config variables that can be logged by splunk log driver.
container-image-name No The container image name. This is part of the docker config variables that can be logged by splunk log driver.
container-env No The container environment variables map in json format. This is part of the docker config variables that can be logged by splunk log driver.
container-labels No The container labels map in json format. This is part of the docker config variables that can be logged by splunk log driver.

Additional log driver options

Amazon CloudWatch Logs

The following additional arguments are supported for the awslogs shim logger binary, which can be used to send container logs to Amazon CloudWatch Logs.

Name Required Description
awslogs-group Yes The log group in which the log stream for the container will be created.
awslogs-stream Yes The log stream name to stream container logs to.
awslogs-region Yes The region name in which the log group and log stream needs to be created in.
awslogs-credentials-endpoint Yes The endpoint from which credentials are retrieved from to connect to Amazon CloudWatch Logs.
awslogs-create-group No Set to false by default. If the provided log group name does not exist and this value is set to false, the binary will directly exit with an error
awslogs-multiline-pattern No Matches the behavior of the awslogs Docker log driver.
awslogs-datetime-format No Matches the behavior of the awslogs Docker log driver

Splunk

The following additional arguments are supported for the splunk shim logger binary, which can be used to send container logs to splunk. You can find a description of what these parameters are used for here.

Name Required
splunk-token Yes
splunk-url Yes
splunk-source No
splunk-sourcetype No
splunk-index No
splunk-capath No
splunk-caname No
splunk-insecureskipverify No
splunk-format No
splunk-verify-connection No
splunk-gzip No
splunk-gzip-level No
splunk-tag No
labels No
env No
env-regex No

Fluentd

The following additional arguments are supported for the fluentd shim logger binary, which can be used to send container logs to Fluentd. Note that all of these are optional arguments.

  • fluentd-address: The address of the Fluentd server to connect to. By default, the localhost:24224 address is used.
  • fluentd-async-connect: Specifies if the logger connects to Fluentd in background. Defaults to false.
  • fluentd-sub-second-precision: Generates logs in nanoseconds. Defaults to true. Note that this is in contrast to the default behaviour of fluentd log driver where it defaults to false.
  • fluentd-tag: Specifies the tag used for log messages. Defaults to the first 12 characters of container ID.

License

This project is licensed under the Apache-2.0 License.

Amazon Ecs Shim Loggers For Containerd

Logger shim repository for streaming container logs when using Containerd

Amazon Ecs Shim Loggers For Containerd Info

⭐ Stars 23
🔗 Source Code github.com
🕒 Last Update a year ago
🕒 Created 2 years ago
🐞 Open Issues 0
➗ Star-Issue Ratio Infinity
😎 Author aws