Journald system and Docker logs
Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.
This setup is useful for those who want to bypass syslog altogether when sending their journald daemon logs or docker container logs using journald docker logging driver. The journald to syslog to loggly approach is not optimal as it loses all the good metadata that journald contains. For example, when docker is forwarding its logs to journald, the CONTAINER_ID and CONTAINER_NAME are captured, which is not the case with the syslog approach.
Note: This setup is supported by systemd so please ensure you are on the latest Linux distribution that supports systemd and systemctl commands to start/stop/restart the system services.
Logging Setup
1. Configuring journald
We are going to use a systemd service which will be responsible for collecting the journald daemon logs and forwarding them to Loggly without excluding all the attached metadata. Please create a service file journald-forwarder.service under your /etc/systemd/system directory by running the command below:
sudo vi journald-forwarder.service
Paste the content below content in the above created file:
[Unit]
Description=Forward journald logs to Loggly
After=docker.service
[Service]
ExecStartPre=-/bin/mkdir -pv /opt/loggly/journald-forwarder
ExecStartPre=-/usr/bin/curl -L -o /opt/loggly/journald-forwarder/journald-forwarder https://github.com/uswitch/journald-forwarder/releases/download/v0.4/journald-forwarder
ExecStartPre=-/bin/chmod +x /opt/loggly/journald-forwarder/journald-forwarder
ExecStart=/opt/loggly/journald-forwarder/journald-forwarder -token [token] -tag journald-loggly
Replace:
- [token]: insert your customer token from the source setup page
2. Reload units and start Journald
Since the journald-forwarder.service changed on disk, you may have to run the command below to reload units:
sudo systemctl daemon-reload
Now, copy and paste the following command in terminal to restart journald-forwarder.service:
sudo systemctl start journald-forwarder.service
You can check the status of the service by running the command below and it must be active and running:
sudo systemctl status journald-forwarder.service
Now your service is started and set up to monitor the journald daemon. It will forward all the logs that are coming to journald. To verify sending system logs run the command below.
logger Hello World
3. Verify Events
Search Loggly over the recent past to find logs with the journald-loggly tag . It may take a few minutes to index them. If you don’t see them, check the troubleshooting section below.
Collecting Docker Containers Logs
You can forward your docker container logs to Loggly by using the journald docker logging driver. The journald logging driver forwards docker container logs to systemd which then forwards them to Loggly using the journald-forwarder service.
Forwarding Nginx docker container logs
Since journald captures important metadata, we are going to demonstrate how to start a docker container using the jouranld logging driver and generate nginx server events to send them to Loggly.
At this point, we assume that you have your journald-forwarder service running already. Please run the command below to start your Nginx container which will be running on port 8080.
docker run --log-driver=journald -d -p 8080:80 nginx
You can see that we used the journald driver and it is exposed on port 8080. You can verify the running container by using the below command:
docker ps -a
Now generate some server logs by hitting the localhost URL below:
curl https://localhost:8080/testing
You can see the combined output of the above commands in the snippet below:
root@ip-172-31-48-18:/home/ubuntu# docker run --log-driver=journald -d -p 8080:80 nginx b9b7f5b335d485b512e11653cb5bc89c5de30087ac4f4510fed088bbd9b9d1c5 root@ip-172-31-48-18:/home/ubuntu# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b9b7f5b335d4 nginx "nginx -g 'daemon of…" 5 seconds ago Up 5 seconds 0.0.0.0:8080->80/tcp practical_kare root@ip-172-31-48-18:/home/ubuntu# curl https://localhost:8080/testing
Verify Events
Search Loggly over the recent past for logs with the journald-loggly tag . It may take a few minutes to index them. If you don’t see them, check the troubleshooting section below.
In the event shown above, you can clearly see all the attached metadata related to the docker container like CONTAINER_NAME and CONTAINER_ID as well as the machine.
Advanced Options
- You can confirm that your docker container is using the journald logging driver by running the command below:
docker inspect -f '{{.HostConfig.LogConfig.Type}}' container_nameReplace:
- container_name: enter your docker container name.
The output of the above command will be similar to the following:
root@ip-172-31-48-18:/home/ubuntu# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b9b7f5b335d4 nginx "nginx -g 'daemon of…" About an hour ago Up About an hour 0.0.0.0:8080->80/tcp practical_kare root@ip-172-31-48-18:/home/ubuntu# docker inspect -f '{{.HostConfig.LogConfig.Type}}' practical_kare journald root@ip-172-31-48-18:/home/ubuntu#
- Since we are using the journald logging driver, which supports the journalctl command to see the container’s logs locally and for many other debugging purposes, we can also check our nginx container’s logs locally by running the command below:
journalctl CONTAINER_NAME=container_nameReplace:
- container_name: enter your docker container name.
Optional: You can also put your container’s ID in place of the container name.
See the example below:
root@ip-172-31-48-18:/home/ubuntu# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b9b7f5b335d4 nginx "nginx -g 'daemon of…" 34 minutes ago Up 34 minutes 0.0.0.0:8080->80/tcp practical_kare root@ip-172-31-48-18:/home/ubuntu# journalctl CONTAINER_NAME=practical_kare -- Logs begin at Fri 2018-03-23 12:32:24 UTC, end at Thu 2018-03-29 12:25:16 UTC. -- Mar 29 11:53:31 ip-172-31-48-18 b9b7f5b335d4[4116]: 2018/03/29 11:53:31 [error] 5#5: *1 open() "/usr/share/nginx/html/testing" failed (2: No such file or directory), cl Mar 29 11:53:31 ip-172-31-48-18 b9b7f5b335d4[4116]: 172.17.0.1 - - [29/Mar/2018:11:53:31 +0000] "GET /testing HTTP/1.1" 404 170 "-" "curl/7.47.0" "-" Mar 29 12:03:40 ip-172-31-48-18 b9b7f5b335d4[4116]: 2018/03/29 12:03:40 [error] 5#5: *2 open() "/usr/share/nginx/html/testing" failed (2: No such file or directory), cl Mar 29 12:03:40 ip-172-31-48-18 b9b7f5b335d4[4116]: 172.17.0.1 - - [29/Mar/2018:12:03:40 +0000] "GET /testing HTTP/1.1" 404 170 "-" "curl/7.47.0" "-" root@ip-172-31-48-18:/home/ubuntu#
Troubleshooting Journald
- Make sure your journald-forwarder service is active and running.
- Make sure you have replaced your customer token in the journald-forwarder.service file correctly.
- Search or post your own question in the community forum.
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.