Troubleshooting Rsyslog

Use these tips to troubleshoot problems with Rsyslog. You can use our automated test, check the configuration, send sample data, and check transmission. Additionally, you can read the Rsyslog manual, try their support forum (they offer professional Rsyslog support), or check out our Rsyslog manual configuration docs.

Wait a Few Minutes

Wait a few minutes after sending an event to give it time to index and appear in the search results. It normally happens within seconds, but sometimes it can take longer.

Check Loggly Status

If Loggly isn’t seeing data check our status page to make sure we are indexing data and search is running. You should see green dots and "All Systems Operational".

Automated Verification

Our configure-syslog script can send a test event to Loggly, and then verify if it’s received using the Loggly search API. You can overwrite your existing loggly configuration to make sure there are no errors and verify it again. It may take a few minutes to run.

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME

Replace:

• SUBDOMAIN: your account subdomain that you created when you signed up for Loggly
• USERNAME: your Loggly username

Check Rsyslog Configuration

Make sure you restarted rsyslog so your changes take effect

sudo service rsyslog restart 

Make sure rsyslog is running. If this command returns nothing than it’s not running.

ps -A | grep rsyslog 

Check the rsyslog configuration. If there are no errors listed, then it’s ok.

rsyslogd -N1 

Make sure you have Rsyslog version 5.8 or higher

rsyslogd -version 

Check the Linux system log for rsyslog errors. You should see an event that it started and no errors. Some logs may also be in /var/log/syslog.

sudo cat /var/log/messages | grep rsyslog 

Make sure Loggly is configured in your rsyslog configuration. There should be an endpoint for logs-01.loggly.com either in your main rsyslog.conf file or an include to the 22-loggly.conf file.

sudo vim /etc/rsyslog.d/22-loggly.conf 

Check the permissions of rsyslog and the file you want to monitor to be sure it can read that file. You may need to use alter the privilege in the rsyslog.conf file:

$PrivDropToUser adm 

Send Sample Data

Verify rsyslog is sending data to Loggly by making a test event. Then search for that event in Loggly by searching for "TroubleshootingTest" in the last hour.

logger TroubleshootingTest 

Check the Linux system log to see if Rsyslog recorded the test event

sudo cat /var/log/messages | grep TroubleshootingTest 

If you are sending repeated test messages, you should turn off repeated message reduction in the rsyslog configuration.

$RepeatedMsgReduction off 

If you are filtering events out with a lower priority, you should send test events with a high enough priority.

logger -p local0.error "TroubleshootingTest" 

Check Data Transmission

Use netstat to verify Rsyslog has an established connection to Loggly. Specifically, check that Loggly can make a connection through your firewall on the proper port. It’s 514 for syslog, 6514 for TLS syslog, 80 for HTTP, and 643 for HTTPS.

sudo netstat -taupn | grep syslog 

Use telnet to verify we can make an outbound connection to Loggly. For syslog we use port 514. If you can’t connect it might be a network or firewall issue.

telnet logs-01.loggly.com 514 

Use tcpdump to verify data is being sent to Loggly. If you send your events in cleartext while tcpdump is running, you should be able to see them in the left hand column.

sudo tcpdump -A dst logs-01.loggly.com 

If your application logs syslog to rsyslog, you can also test to see if messages making it to rsyslog over UDP to localhost.

sudo tcpdump -i lo -A udp and port 514 

Check Log Rotation

Some older version of rsyslog may have trouble resuming after a log is rotated. If you have log rotation setup, follow these instructions to force rsyslog to pick up the new file.

Read More

• Rsyslog-users – Mailing list for rsyslog describing many common support issues

Still Not Working?

Please search our community forum for more Rsyslog configuration answers or post your own question.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.