Documentation forLoggly

Docker logging through Logspout

Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.

You can send Docker container logs to Loggly with the help of Logspout. This setup doesn’t support multiline logs, each line will be treated as a separate log event. This setup allows sending logs from multiple containers to a centralized guest container from which Logspout can then send to Loggly. All the logs from the guest container will be forwarded to Logspout automatically until the guest container is restarted or exited, in which case, you will have to repeat the procedure again for centralizing logging. This setup has been tested with Docker version 17.03.2-ce and Logspout version 3.2.6.

Docker Logspout Setup

Run the Logspout container

Run the command below to start the Logspout container and configure it to send logs to Loggly.

docker run --name logspout -d --volume=/var/run/docker.sock:/var/run/docker.sock -e SYSLOG_STRUCTURED_DATA="TOKEN@41058 tag=\"Logspout\"" gliderlabs/logspout syslog+tcp://logs-01.loggly.com:514

Replace:

  • TOKEN: enter your customer token from the Source Setup page in Loggly

Verify Events

Go to the Search tab in Loggly and search for events with the Logspout tag over the last 20 minutes. It may take a few minutes to index the events. If it doesn’t work, see the troubleshooting section below.

tag:Logspout

Docker Syslog Example

Note: The container name and ID of the container will be parsed as syslog.appname and syslog.host respectively.

Centralization of logs from multiple containers

If you want to send multiple container logs to Loggly, you don’t need to set up each container to send the logs. The already running Logspout container will take care of other container’s logs and will forward them with its own container logs to Loggly.

"The logspout container has the ability to gather logs from other containers that are started without the -t option and are configured with a logging driver that works with docker logs (journald and json-file)."

Example for centralizing nginx server logs with Logspout

Step 1. You should  already have started the Logspout container and you can see the running container status by running the command below:

root@ip-172-31-57-234:/home/ubuntu# docker ps -a
CONTAINER ID    IMAGE                COMMAND                 CREATED          STATUS          PORTS        NAMES
4280a44b3a0a    gliderlabs/logspout  "/bin/logspout sys..."  3 minutes ago    Up 3 minutes    80/tcp       logspout
root@ip-172-31-57-234:/home/ubuntu#

Step 2. Simply Run the nginx container using the command below.

docker run --name nginx -d -p 8080:80 nginx

You can now see that both Logspout and nginx containers are running.

root@ip-172-31-57-234:/home/ubuntu# docker ps -a
CONTAINER ID    IMAGE                COMMAND                 CREATED         STATUS          PORTS                    NAMES
9f6e5e63fb4c    nginx                "nginx -g 'daemon ..."  1 second ago    Up 1 second     0.0.0.0:8080->80/tcp  nginx
4280a44b3a0a    gliderlabs/logspout  "/bin/logspout sys..."  4 minutes ago   Up 4 minutes    80/tcp                   logspout
root@ip-172-31-57-234:/home/ubuntu#

Step 3. Your nginx container has started and is open to listen at port 8080. Now you are all set to generate nginx server logs by hitting the localhost URL with the 8080 port and logspout will ship all the logs to Loggly.

Verify Events

Search Loggly for events with the Logspout tag over the last 20 minutes. It may take a few minutes to index the events. If it doesn’t work, see the troubleshooting section below.

tag:Logspout

You should see tag: Logspout which shows that nginx server logs are coming from the Logspout container. The logtype is nginx because the logs are nginx. You will also notice the container name and ID in syslog.appName and syslog.host, respectively. You can easily differentiate between different containers and their logs by looking at the container IDs.

Multiline Support

In order to enable multiline logging, you must first prefix your adapter with the multiline adapter as shown below:

docker run --name logspout -d --volume=/var/run/docker.sock:/var/run/docker.sock -e SYSLOG_STRUCTURED_DATA="TOKEN@41058 tag=\"Logspout\"" gliderlabs/logspout multiline+syslog+tcp://logs-01.loggly.com:514

Replace:

  • TOKEN: enter your customer token from the Source Setup page in Loggly

Adding multiline adapter will allow you to see your multiline events like error stacktraces into a single event. Please see the screenshot below  to get an idea of it.

Note: If you are not using the multiline adapter and logging a multiline event or error stacktrace then you will see each line of your stacktrace as a single event in Loggly.

Advanced Options

Docker Logging Troubleshooting

If you don’t see any data show up in the verification step, check for these common problems.

Check Docker Container:

  • Wait a few minutes in case indexing needs to catch up
  • Verify the container is running by running sudo docker ps -a
  • Make sure your machine has Rsyslog installed on it
  • Send test events from inside each of the containers and from the host to see which point in the chain is dropping logs
  • You can only relay the other container’s logs to logspout which supports "json-file" or "journald" driver because only "json-file" and "journald" driver supports the "docker logs" command. Read more on limitations-of-logging-drivers

Still Not Working?

  • Search or post your own Docker logs, Docker daemon, or other Docker question in the community forum.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.