Configure Kafka integration

To integrate Kafka with SolarWinds Observability:

1. In SolarWinds Observability, click Add Data at the top.

2. In the Add Data dialog, click Integration, and then click Kafka.

3. In the Add Kafka dialog, select a host with the SolarWinds Observability Agent you want to use for Kafka and click Next.

   To only display Agents running in a Kubernetes cluster, click the List only Agents... toggle.

   If you want to install a dedicated Agent on a dedicated host, add the host and then add the Kafka integration. See Add a self-managed host.

   If you prefer using Docker, see Run the SolarWinds Observability Agent as a Docker image or Kubernetes deployment.

4. Select where the Kafka service is located:

   • On Host: The Kafka service runs on the same physical or virtual machine as the Agent.

   • Off Host: The Kafka service runs on a different physical or virtual machine than the Agent.

5. Install and configure Prometheus JMX Exporter and configure Kafka settings.

   1. Download the Prometheus JMX Exporter from the official github repository.

   2. Download the Kafka configuration file (jmx-kafka-config.yaml) from a SolarWinds server.

      If your JMX connection is protected, or should be done over SSL, open the YAML file and add the following details to the top of the file, before lowercaseOutputName: true.

      For details, see JMX Exporter documentation.

      username: <username>
      password: <password>
      ssl: true
      
      lowercaseOutputName: true
      rules:
      ...

6. Configure the EXTRA_ARGS environment variable for the JMX Exporter configuration.

   1. Review the configuration of the JMX Exporter and adjust the settings if appropriate. The items will be used to construct the code for you to copy and run to configure the EXTRA_ARGS environment variable.

      • Review the URL of the Kafka JMX Exporter endpoint in Kafka Endpoint.

      • Review the location of the JMX Prometheus JAR file in JAR File Location.

      • Review the location of jmx-kafka-config.yaml in Config File Location.

   2. Click Copy to Clipboard and run the code.

      Example:

      EXTRA_ARGS=-javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.18.0.jar=localhost:1234:/jmx-exporter/jmx-kafka-config.yml

      The generated code explained:

      EXTRA_ARGS=-javaagent:<path to the Prometheus JMX Exporter .JAR>=<host>:<port>:<path to the configuration file>

      SolarWinds recommends that you do not configure the JMX exporter to bind to all interfaces. However, you can configure it by specifying only the port number in the environment variable.

      Example:

      EXTRA_ARGS=-javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.18.0.jar=1234:/jmx-exporter/jmx-kafka-config.yml

7. Restart the Kafka server.

8. Run the command to test that Kafka metrics are exposed. The call should return a list of metrics.

   curl http://localhost:1234

9. Click Next.

10. Review the configuration and click Deploy to deploy the changes to the agent.

    • Provide a Display Name.

    • Review the Kafka Endpoint.

    • Review the Collection Interval.

11. When the installation is complete, you can continue to observe Kafka data. Use the buttons on the Summary screen.

To view data collected for the newly configured integration, click Dashboards and search for Kafka. Click the system dashboard for Kafka. See Dashboards.

To view metrics collected for Kafka, click Analyze > Metrics and search for Kafka. See Metrics Explorer for details.