Linux Log File Monitoring

Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.

This guide will help you setup Loggly as your continuous log file monitoring tool. This script configures both files and directories to send logs to Loggly. It will automatically watch files for new logs appended to the end of the file. Once you configured a directory, it automatically sends the newly added files within the directory to Loggly. It assumes you use rsyslog 5.8 or higher, TCP over port 514, and you have sudo permissions. For alternatives, please see the Advanced Options section.

Automatic Script

1.Run The Configure File Monitoring Script

Run our automatic configure-file-monitoring script below to continuously read the files and send the logs to Loggly through your syslog daemon. Alternatively, you can follow our manual configuration instructions below.

curl -O https://www.loggly.com/install/configure-file-monitoring.sh
sudo bash configure-file-monitoring.sh -a SUBDOMAIN -t TOKEN -u USERNAME -f FILENAME -l ALIAS 

Replace:

• SUBDOMAIN: your account subdomain that you created when you signed up for Loggly
• TOKEN: your customer token from the source setup page. You can find your customer token by going to the "Source Setup" section and the "Customer Tokens" subsection as shown below.
  
• USERNAME: your Loggly username, which is visible at the top right of the Loggly console
• FILENAME: the file or directory you want to monitor, can contain wildcards but cannot contain spaces
• ALIAS: an easy to recognize name for the syslog.appName field. Must be unique for each file.

You will need to enter your system root password so it can update your rsyslog configuration. It will then prompt for your Loggly password.

2. Verify Events

Search Loggly for events with the file tag over the past hour. It may take a few minutes to index the event. If it doesn’t work, see the troubleshooting section below.

tag:file 

Manual Configuration

If you prefer, you can configure rsyslog directly. You can monitor files using the manual configuration. However, directory monitoring requires use of the script above.

1. Configure Syslog Daemon

If you haven’t already, run our automatic Configure-Syslog script below to setup rsyslog. Alternatively, you can Manually Configure Rsyslog or Syslog-ng.

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME 

Replace:

• SUBDOMAIN: your account subdomain that you created when you signed up for Loggly
• USERNAME: your Loggly username, which is visible at the top right of the Loggly console

2. Retrieving your Customer Token

Customer token is a universally unique identifier which differentiate your logs from everyone else’s. Upon account set-up we generate a customer token for you. The customer token is an alpha-numeric string that needs to be included with each log event. You will continue to use that token throughout the term of your account for all of your logs unless you have retired the token and created a new token.

After you set-up your account you can see your customer token by going to the "Source Setup" section and the "Customer Tokens" subsection. You can click the copy button to copy it to your clipboard, and then paste it into your source configuration.

3. Setup File Monitoring

Copy this to your terminal window and run it. It will make sure the working directory exists. If it’s an Ubuntu system, it will set the proper permissions. It will then open a file monitoring configuration file.

sudo mkdir -v /var/spool/rsyslog
if [ "$(lsb_release -ds | grep Ubuntu)" != "" ]; then
  sudo chown -R syslog:adm /var/spool/rsyslog
fi
sudo vim /etc/rsyslog.d/21-filemonitoring-loggly.conf
 

Copy in this additional configuration to add file monitoring, replacing the variables below.

If you are using version 7.x or lower, paste this configuration:

$ModLoad imfile
$InputFilePollInterval 10 
$PrivDropToGroup adm

# Input for FILE1
$InputFileName FILE1
$InputFileTag APPNAME1
$InputFileStateFile stat-APPNAME1 #this must be unique for each file being polled
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Add a tag for file events
$template LogglyFormatFile,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"file\"] %msg%\n"

# Send to Loggly then discard
if $programname == 'APPNAME1' then @@logs-01.loggly.com:514;LogglyFormatFile
if $programname == 'APPNAME1' then ~ 

If you are using version 8.x or higher, paste this configuration:

module(load="imfile")

# Input for FILE1
input(type="imfile" tag="APPNAME1" ruleset="filelog" file="/FILE1") #wildcard is allowed at file level only

# Add a tag for file events
template(name="LogglyFormatFile" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"file\" ] %msg%\n")

ruleset(name="filelog"){
  action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="514" template="LogglyFormatFile") stop
} 

Replace:

• FILE1: the name of the file you’d like to monitor, or multiple files using a wildcard (*). For example, /home/ubuntu/myapp/logs/*
• APPNAME1: an application name for the file
• TOKEN: your customer token from the source setup page as shown in the step 2 above.

NOTE: Wildcard is only supported on 8.x or higher versions and are supported at the file level, not at the directory level. If you have 7.x or lower you have to use cron generated by the script.

Restart rsyslogd

$ sudo service rsyslog restart 

4. Verify Events

Search Loggly for events with the file tag tag over the past hour. It may take events a few minutes to index. If it doesn’t work, see the troubleshooting section below.

tag:file 

Advanced Log File Monitoring Options

• Upload a single log file – to do a quick test by uploading a single file using curl
• Windows File Monitoring – monitor a file on Windows systems
• Linux File Monitoring – monitor a file on Linux systems
• Script Source on GitHub – View the source or suggest improvements
• Log Rotation – if you use log rotation, you must add commands to refresh the files
• NOTE: For wildcard, the kernel should support inotify. If your kernel does not support inotify you have to include $InputFilePollInterval 10 in 8.x configuration
• Search or post your own Windows, Linux, or general log file monitoring questions in the community forum.

Log File Monitoring Troubleshooting

If you don’t see any data show up in the verification step, then check for these common problems.

Check File Monitoring Config:

• Wait a few minutes in case indexing needs to catch up
• If you followed the manual instructions, make sure you replaced your customer token, file name, and app name tag in the configuration file. There may be up to 4 instances of APPNAME1 that need to be replaced.
• Check the log files to make sure they exist and you have the right path

Check Your Syslog Daemon:

• Troubleshooting Rsyslog

SELinux Error:

• Sample Error Messages:

  ERROR: selinux status is 'Enforcing'. Please manually restart the rsyslog daemon or turn off selinux by running 'setenforce 0' and then rerun the script. 

  If you see this error then you will need to disable the SELinux by setting it to permissive mode using the below command:

  sudo setenforce 0 

Still Not Working?

• Search or post your own question in the community forum.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.