Documentation forLoggly

Log Rotate Configuration

Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.

Some older versions of rsyslog may have trouble resuming on a log file after the log rotate has run. We can force rsyslog to pick up the new log file by adding a postrotate command in logrotate. This will restart rsyslog and delete the state files so it will continue reading from the beginning of the newly created file.

Log Rotate Setup

1. Open the logrotate configuration file

Configuration files are located in this directory on most linux distributions

cd /etc/logrotate.d

Find the appropriate configuration file and open it with a text editor. In this example, we’ll use the nginx log files.

sudo vim nginx

You will see the default configuration given below

/var/log/nginx/*.log {
  daily
  missingok
  rotate 52
  compress
  delaycompress
  notifempty
  create 640 nginx adm
  sharedscripts
  postrotate
  [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
  endscript
}

All the commands written between the postrotate and endscript gets executed after each log rotation. In this example, we can see that nginx is restarted. We will add additional commands here soon.

2. Find your rsyslog state files

Find the files rsyslog writes to track the state of the files are monitoring. We will configure the postrotate command to delete these configuration files so rsyslog starts fresh at the beginning of the new log file. If you used the configure-file-monitoring script, it will include the alias you passed as a parameter.

ls /var/spool/rsyslog/stat-*

Here we can see two state files for nginx

/var/spool/rsyslog/stat-nginx-access
/var/spool/rsyslog/stat-nginx-error

3. Add postrotate commands

Add the following commands in the postrotate section to restart rsyslog and delete the state files. Replace the path given to the rm command with the path to the rsyslog state files found above. In this example, we are deleting the state files for nginx.

rm /var/spool/rsyslog/stat-*
service rsyslog stop
service rsyslog start

Advanced Log Rotate Options

Troubleshooting Your Log Rotate Configuration

If you don’t see any data show up in the verification step, then check for these common problems.

  • Wait a few minutes in case indexing needs to catch up
  • If you see duplicate events send to Loggly, check to see if you accidentally deleted the wrong state files. Also check to make sure a new file of zero length is created after the log rotation runs.
  • Make sure the state files are deleted and recreated after rsyslog restarts
  • Troubleshooting Rsyslog if the files are being written but not being sent to Loggly
  • Search or post your own question in the community forum.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.