HAProxy logging using syslog

Loggly provides the infrastructure to aggregate and normalize log events so they are available to explore interactively, build visualizations, or create threshold-based alerting. In general, any method to send logs from a system or application to an external source can be adapted to send logs to Loggly. The following instructions provide one scenario for sending logs to Loggly.

This document provides an overview of the features and benefits of using load balancing with HAProxy. Load balancing provides better performance, availability, and redundancy because it spreads work among many back-end servers. This will:

Reduce the load on each individual machine.
Let you do maintenance, upgrade software, deploy websites, or swap out servers without disrupting service.
Enable your website or service to handle more traffic.
Increase overall performance.

HAProxy is a free, open-source reverse proxy and load balancer with the ability to handle hundreds of thousands of simultaneous connections. It has several features which allow it to work well with web traffic, such as its ability to inspect and direct clients based on their HTTP messages.

Step 1. Configure Syslog Daemon

Run our automatic Configure-Syslog script below to set up rsyslog. Alternatively, you can Manually Configure Rsyslog or Syslog-ng.

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a SUBDOMAIN -u USERNAME

Replace:

SUBDOMAIN: enter the account subdomain that you created when you signed up for Loggly
USERNAME: enter your Loggly username, which is visible at the top right of the Loggly console

You will need to enter your system root password so it can update your rsyslog configuration. It will then prompt for your Loggly password.

Step 2. Install and Start HAProxy

Install the haproxy package with following command:

sudo apt-get install haproxy

After installation, verify that HAProxy is working:

haproxy -v

The server will respond with:

HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>

You can also find created files in the following locations:

/etc/default/haproxy
/etc/haproxy
/etc/init.d
/etc/logrotate.d/haproxy
/run/haproxy
/usr/sbin
/usr/share/lintian/overrides/haproxy
/var/lib/haproxy

The HAProxy main configuration file is located at:

/etc/haproxy/haproxy.cfg

The above file starts a server with the default settings and below is the content of this file:

global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  # Default ciphers to use on SSL-enabled listening sockets.
  # For more information, see ciphers(1SSL). This list is from:
  # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  ssl-default-bind-options no-sslv3
defaults
  log   global
  mode  http
  option httplog
  option dontlognull
  timeout connect 5000
  timeout client 50000
  timeout server 50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

The global section contains settings that apply to the HAProxy process itself. This section contains user, group, log directives, and stats. The defaults section contains all of the proxies settings.

After editing /etc/haproxy/haproxy.cfg, restart the haproxy service for the changes to take effect:

sudo service haproxy restart

Step 3. Setting Up Required Configurations and Load Balancing

Now let’s put HAProxy in front of your web server(s).

At the bottom of the /etc/haproxy/haproxy.cfg file, we will add one or more new sections. In addition to global and defaults you can add:

frontend: defines a reverse proxy which will listen for incoming requests on a specific IP address and port.
backend: defines a pool of servers that the frontend will forward requests to.
listen: a shorthand notation that combines frontend and backend features into a single command.

Let’s define a frontend first. We will have it listen on the HAProxy IP address at port 80. Open /etc/haproxy/haproxy.cfg configuration file:

sudo vi /etc/haproxy/haproxy.cfg

Add the following lines to the bottom of the /etc/haproxy/haproxy.cfg file:

frontend firstbalance
  bind *:80
  option forwardfor
  default_backend webservers

You can use either an IP address or an asterisk *, which means any IP address configured on this machine. You can add as many bind directives to frontend as you want.

We also want to capture the client’s source IP address in our web server’s logs. We don’t want to log our proxy server’s IP address. To do this, we added a header called forwardfor to the incoming request before it is passed along to the web servers. Then the web server can look for and parse this header to get the original, client IP address.

We can also add an optional forwardfor directive to the defaults section so that it applies across the board to all proxies.

Our frontend section is configured, via its default_backend directive, to forward requests to a pool of servers defined in a backend section called webservers. We can distribute the load across many servers. We just add more server directives. webservers backend shares the traffic equally among two web servers by using the roundrobin algorithm.

The roundrobin load-balancing algorithm is the default if you don’t set the balance directive. A balance directive can be set in a backend section or globally in the defaults section. If it’s set in the defaults section, it can be overridden within a backend to use a different algorithm. webserver1 would get the first request. The second would go to webserver2. The third would go back to webserver1, etc. We can add as many server directives as we want to a single backend.

We enabled health checking by adding the check parameter to the webserver1 and webserver2 servers directive. A drawback of enabling health checks with the check parameter is that success is measured only by the ability to make a TCP connection to the back-end servers IP address and port. So, even if the web servers begins responding with HTTP 500 Internal Server Error, the checks will still pass as long as a connection can be made. So, visitors might be sent to web pages that are displaying errors. We are using mode http in default section. We can have them look for a successful HTTP response. Any response between the 2xx-3xx range is good. To use this feature, we added the option httpchk directive to the back-end.

Let’s go ahead and define that. Add the following lines below the frontend section:

backend webservers
balance roundrobin
server webserver1 Your-Webserver1-IP:80 check
server webserver2 Your-Webserver2-IP:80 check
option httpchk

Replace:

Your-Webserver1-IP: enter your own webserver IP

Save changes and close the /etc/haproxy/haproxy.cfg file.

We can verify that the configuration file is valid with the following command. If there are any errors, they will be displayed so that we can go in and fix them:

haproxy -f /etc/haproxy/haproxy.cfg -c

Now that the necessary changes have been made, restart the haproxy service:

sudo service haproxy restart

Step 4. Generate Events

Now you can create some HTML/PHP files on the web servers for testing the HAProxy load balancer. When you make a request to the load balancer’s IP address, you should see your HTML or PHP web content.

For example, if I hit the localhost URL, as shown below, it will generate and send the event to Loggly.

curl https://localhost:80/First

Step 5. Verify Events

Search Loggly for events with the RsyslogTLS tag over the past hour. It may take a few minutes to index the event. If it doesn’t work, see the troubleshooting section below.

Advanced HAProxy configurations Options

If you want, you can use a single listen section as an alternative. The listen section uses a single listen directive instead of using both a frontend and backend section. The listen section combines the features of both. To implement that, you should remove both the existing frontend and backend section from the /etc/haproxy/haproxy.cfg file and replace them with the configuration below at the bottom of the file:
```
listen firstbalance
  bind *:80
  balance roundrobin
  option forwardfor
  option httpchk
  server webserver1 Your-Webserver1-IP:80 check
  server webserver2 Your-Webserver2-IP:80 check 
```
If you would like to monitor live traffic that passes through HAProxy, enable debugging with the -d flag:
```
haproxy -f /etc/haproxy/haproxy.cfg -d 
```

HAProxy Logs Troubleshooting

If you don’t see any data show up in the verification step, then check for these common problems.

Wait a few minutes in case indexing needs to catch up
Try manually configuring rsyslog if the script doesn’t work
See our Rsyslog Troubleshooting Guide
Check that you are using one of our automatically parsed formats
Search or post your own question in the community forum.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.

Search SolarWinds Support