Documentation forLoggly

Scrub Sensitive Data in Rsyslog

When your company has data that it should not expose due to concerns about security or privacy, you can scrub or mask the data from the logs. This removes the sensitive information before it leaves your network. Alternatively, we offer role based access control if there is sensitive information you prefer to keep in the logs, and you want to control who has access. The following example will scrub a 16 Digit Credit Card Number from the logs, but you can scrub any string that follows a regular expression pattern. This example requires Rsyslog version 8.x or higher.

Rsyslog Setup

1. Configure Syslog Daemon

If you haven’t already, run our automatic Configure-Syslog script below to setup rsyslog. Alternatively, you can Manually Configure Rsyslog or Syslog-ng.

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME 

Replace:

  • SUBDOMAIN: your account subdomain that you created when you signed up for Loggly
  • USERNAME: your Loggly username, which is visible at the top right of the Loggly console

You will need to enter your system root password so it can update your rsyslog configuration. It will then prompt for your Loggly password.

2. Update Configuration

Copy and replace the below mentioned LogglyFormat Template in /etc/rsyslog.d/22-loggly.conf

$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058] %$!msg%n" 

Replace:

Add the following code below LogglyFormat Template in same file. This example will match 16 Digit Credit Card number. Please substitute your own regular expression to meet your own requirements.

if re_match($msg,'(5[1-5][0-9]{14})') 	
then 
{
  set $!ext = re_extract($msg,'(5[1-5][0-9]{14})',0,1,"");
  set $!msg= replace($msg, $!ext, "xxxxxxxxxxxxxxxx");
}	
else 
  set $!msg = $msg;
 

Save the file and restart rsyslog.

 sudo service rsyslog restart 

3. Send A Test Event

Use Logger to send a test event to Loggly. In the example below we are sending one sample credit card number.

logger 'credit card number is 5255224165541111'

4. Verify

Search Loggly over the past 10 minutes to find your logs. It may take a few minutes to index them. Click on one of the logs to show a list of syslog fields along with scrubbed info. If you don’t see them, check the troubleshooting section below.
Scrubbed log scrub

Advanced Rsyslog Configuration Options

Troubleshooting Linux Syslog

  • The rsyslog versions supported for this example are 8.x or higher as CEE Lumberjack properties are not supported by earlier versions.
  • Make change in template where we are declaring a new variable %$!msg%
  • The regex which you are using in the script could be invalid rsyslog regex. Please test the regex and input string.
  • Try manually configuring rsyslog if the script doesn’t work
  • See our Rsyslog Troubleshooting Guide
  • Search or post your own question in the community forum.

When the APM Integrated Experience is enabled, Loggly shares a common navigation and settings with the other integrated experiences' products. How you navigate Loggly and access its features may vary from these instructions. For more information, go to the APM Integrated Experience documentation.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.