Rsyslog TLS Configuration
Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.
You can securely send your logs to Loggly using TLS encryption. This guide shows you how to set it up for rsyslog and is tested on Ubuntu 12.04. For distributions based on Red Hat, please use rpm or yum in place of apt-get. Please see here for information on Rsyslog manual configuration.
Rsyslog TLS Configuration Setup
1. Install rsyslog-gnutls package
Install rsyslog-gnutls package.
sudo apt-get install rsyslog-gnutls
2. Update Configuration file
Open up the configuration file (22-loggly.conf) and update the content by replacing the configuration given below.
vim /etc/rsyslog.d/22-loggly.conf
If you are using version 7.x or lower, paste in this configuration:
########################################################## ### RsyslogTemplate for Loggly ### ########################################################## $template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"RsyslogTLS\"] %msg%\n" # Setup disk assisted queues $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down #RsyslogGnuTLS $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt $ActionSendStreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.loggly.com *.* @@logs-01.loggly.com:6514;LogglyFormat #################END CONFIG FILE#########################
If you are using version 8.x, paste in this configuration:
# Setup disk assisted queues $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down #RsyslogGnuTLS $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt template(name="LogglyFormat" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"RsyslogTLS\"] %msg%\n" ) # Send messages to Loggly over TCP using the template. action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
Replace:
- TOKEN: your customer token from the source setup page
Open your /etc/rsyslog.conf and modify the $MaxMessageSize with
$MaxMessageSize 64k
If $MaxMessageSize is not present then add it in your configuration. For this to work perfectly, add it in the top of the configuration file.
3. Restart Rsyslog
Restart Rsyslog so the changes take effect.
sudo service rsyslog restart
4. Verify Events
Search Loggly for events with the tag as RsyslogTLS over the past hour. It may take few minutes to index the event. If if doesn’t work, see the troubleshooting section below.
tag:RsyslogTLS
Advanced Rsyslog TLS Configuration Options
- Rsyslog Automatic Logging Setup – configuring rsyslog by using a script.
- Rsyslog Manual Configuration – if automatic doesn’t work.
- Syslog-ng TLS Configuration – sending syslog-ng logs using TLS Encryption.
- NXLog TLS Configuration – – sending logs using TLS Encryption.
- The default maximum supported message size is 2KB. Use $MaxMessageSize parameter in /etc/rsyslog.conf file to handle more than 2KB message size.
- Search or post your own rsyslog TLS configuration questions in the community forum.
Troubleshooting Your Rsyslog TLS Configuration
- Wait a few minutes in case indexing needs to catch up
- Make sure you restarted rsyslog
- Syslog over TLS uses port 6514, so check that you’ve updated your rsyslog configuration
- Make sure port 6514 outbound is open on your firewall and network settings
- Verify your operating system has support for TLS 1.1 or 1.2. Older OSes like CentOS 5 do not have support, but CentOS 6 and higher do.
- If you get an error saying "rsyslogd:not permitted to talk to peer, certificate invalid: signer not found" then make sure you concatenated the loggly certificate
- Troubleshooting Rsyslog if the files are being written but not being sent to Loggly
- Search or post your own Rsyslog TLS questions in the community forum.
FAQ
Do I need to change anything right now?
No, but you will need to update your configuration in the upcoming future to ensure that the CA root certificate remains valid (it has its own expiration date). We strongly advise to use the system trust store of CA certificates usually located in /etc/ssl/certs/ca-certificates file.
What to do when my logs are not being sent?
- Check if there are SSL verification errors on the client side.
- If you are using Rsyslog follow our documentation to change the $DefaultNetstreamDriverCAFile to the $DefaultNetstreamDriverCAFile
/etc/ssl/certs/ca-certificates.crt - Restart rsyslog
Why do I need to update my configuration anyway?
The client has to always provide an anchor for verification of the last part of the SSL certificate chain (the CA Root certificate itself). This was so far provided by us in the form of the whole certificate chain that the customer needed to download and configure his systems with. Now it should be provided by the OS certificate store instead so this information is always up to date and dependent only on the update cycle of the customers machines. You can make sure you have the certificates updated by installing e.g. on Debian/Ubuntu systems a package "ca-certificates".
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.