Documentation forAppOptics

Monitoring Kubernetes with AppOptics

AppOptics provides both monitoring for Kubernetes as a cluster, as well as the containers and processes running inside it. Monitoring applications running in Kubernetes is done via our APM agents. To learn more about Kubernetes plugin please see its documentation.

Setup

There are 3 basic ways to monitor a Kubernetes cluster with AppOptics: as a pod, "bare-metal" on the Kubernetes master, and as a sidecar.

Pod setup

In this setup, the SolarWinds Snap Agent is running in a Kubernetes pod.

  • First, you'll need to set up Kubernetes service authentication. To create an SolarWinds service account in the Kubernetes cluster:

    Copy
    $ git clone https://github.com/solarwinds/solarwinds-snap-agent-docker
    $ cd solarwinds-snap-agent-docker
    $ kubectl create -f swisnap-agent-serviceaccount.yaml

    (See solarwinds-snap-agent-docker for more details.)

  • Next, use the deployment manifest defined in solarwinds-snap-agent-docker:

    • Update the manifest to contain your SOLARWINDS_TOKEN
    • In the env section of swisnap-agent-k8s deployment, you can configure which plugins should be run by setting SWISNAP_ENABLE_<plugin_name> to either true or false. Plugins turned on via environment variables are using default configuration and taskfiles.
  • After configuring deployment to your needs (please refer to Integrating AppOptics Plugins with Kubernetes), type and run following command:

    Copy
    $ kubectl create -f swisnap-agent-deployment.yaml
  • Finally, check if the deployment is running properly:

    Copy
    $ kubectl get deployment swisnap-agent-k8s

"Bare-metal" setup

In this setup, the SolarWinds Snap Agent is running on the Kubernetes master.

  • First, the solarwinds user must have access to the kubeconfig.

    • By default, Kubernetes config is stored in /etc/kubernetes/admin.conf
    • Copy the Kubernetes configuration file to destination specified in Kubernetes plugin config and change its owner to solarwinds user
    Copy
    $ sudo mkdir -p /opt/SolarWinds/Snap/etc/.kube
    $ sudo cp /etc/kubernetes/admin.conf /opt/SolarWinds/Snap/etc/.kube/config
    $ sudo chown -R solarwinds:solarwinds /opt/SolarWinds/Snap/etc/.kube
    • Example plugin config:
    Copy
    collector:
    kubernetes:
        all:
        incluster: false
        kubeconfigpath: "/opt/SolarWinds/Snap/etc/.kube/config"
        interval: 60s
    load:
        plugin: snap-plugin-collector-aokubernetes
        task: task-aokubernetes.yaml
    • Note: the plugin config must include the incluster: false setting, which denotes that the SolarWinds Snap Agent is not running as pod in Kubernetes cluster
  • You can easily interact with SolarWinds Snap Agent via swinsap CLI. For additional information about swisnap CLI please see Troubleshooting.

    Copy
    $ swisnap task list

Sidecar setup

In this setup, the agent will monitor only services running in particular pod(s), not Kubernetes itself.

  • Useful when you want to monitor only specific per-pod-services
  • Configuration is similar to pod setup
  • In order to monitor specific services only, the kubernetes and aosystem plugins should be disabled by setting SWISNAP_ENABLE_KUBERNETES to false and SWISNAP_DISABLE_HOSTAGENT to true in swisnap-agent-deployment.yaml

Example: Running Apache Server with sidecar SolarWinds Snap Agent

In order to monitor Apache with the agent in a sidecar, Apache pod config should be passed to swisnap-agent-deployment.yaml or vice-versa, e.g. (notice SWISNAP_ENABLE_APACHE):

Copy
containers:
- name: apache
    imagePullPolicy: Always
    image: '<your-image>'
    ports:
    - containerPort: 80
- name: swisnap-agent-ds
    image: 'solarwinds/solarwinds-snap-agent-docker:1.0.0'
    imagePullPolicy: Always
    env:
    - name: SOLARWINDS_TOKEN
        value: 'SOLARWINDS_TOKEN'
    - name: APPOPTICS_HOSTNAME
        valueFrom:
        fieldRef:
            fieldPath: spec.nodeName
    - name: SWISNAP_ENABLE_DOCKER
        value: 'false'
    - name: SWISNAP_ENABLE_APACHE
        value: 'true'
    - name: SWISNAP_DISABLE_HOSTAGENT
        value: 'true'
    - name: HOST_PROC
        value: '/host/proc'

In the example above, the sidecar will run only the Apache plugin. Additionally, if the default Apache Plugin configuration is not sufficient, custom one should be passed to pod running SolarWinds Snap Agent - Integrating AppOptics Plugins with Kubernetes.

Containers inside the same pod can communicate through localhost, so there's no need to pass a static IP - Resource sharing and communication

Integrating AppOptics Plugins with Kubernetes

In case of "bare-metal" setup please follow steps described in Integrations. When SolarWinds Snap Agent is running inside Kubernetes Pod, integrating AppOptics Plugins with Kubernetes requires some additional steps. SolarWinds Snap Agent image is using default plugins configuration files and tasks manifests. In order to use your own configuration you would have to create Kubernetes configMap. In this example we’ll set up two configMaps, first for plugins and second for tasks.

Copy
# create plugins configMap
kubectl create configmap plugin-configs --from-file=/path/to/my/plugins.d/ --namespace=kube-system
# create tasks configMap
kubectl create configmap task-manifests --from-file=/path/to/my/tasks.d/ --namespace=kube-system
# check if everything is fine
kubectl describe configmaps plugin-configs task-manifests

ConfigMaps should be attached to SolarWinds Snap Agent deployment. Here's the example, notice spec.template.spec.containers.volumeMounts and spec.template.spec.volumes:

Copy
kind: Deployment
apiVersion: apps/v1beta2
metadata:
  name: swisnap-agent-k8s
  namespace: kube-system
  labels:
    app: swisnap-agent-k8s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: swisnap-agent-k8s
template:
  metadata:
    labels:
      app: swisnap-agent-k8s
  spec:
    serviceAccountName: swisnap-agent-serviceaccount
    containers:
      - name: swisnap-agent-k8s
        image: 'solarwinds/solarwinds-snap-agent-docker:1.0.0'
        imagePullPolicy: Always
        volumeMounts:
          - name: plugins-vol
            mountPath: /opt/SolarWinds/Snap/etc/plugins.d
          - name: tasks-vol
            mountPath: /opt/SolarWinds/Snap/etc/tasks.d
        env:
          - name: SOLARWINDS_TOKEN
            value: 'SOLARWINDS_TOKEN'
          - name: SWISNAP_DISABLE_HOSTAGENT
            value: 'true'
        resources:
          limits:
            cpu: 100m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
    volumes:
      - name: plugins-vol
        configMap:
          name: plugin-configs
          items:
            - key: <first-plugin-name>.yaml
              path: <first-plugin-name>.yaml
            - key: <second-plugin-name>.yaml
              path: <second-plugin-name>.yaml
            - key: <third-plugin-name>.yaml
              path: <third-plugin-name>.yaml
      - name: tasks-vol
        configMap:
          name: task-manifests
          items:
            - key: task-ao<first-plugin-name>.yaml
              path: task-ao<first-plugin-name>.yaml
            - key: task-ao<second-plugin-name>.yaml
              path: task-ao<second-plugin-name>.yaml
            - key: task-ao<third-plugin-name>.yaml
              path: task-ao<third-plugin-name>.yaml
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

When you're attaching taskfiles and plugin configuration files through configMaps, there's no need to set environment variables SWISNAP_ENABLE_<plugin-name>. SolarWinds Snap Agent will automatically load plugins based on files stored in configMaps. For further informations please visit solarwinds-snap-agent-docker.

Integrating Kubernetes Cluster Events Collection With Loggly

  • Create kubernetes.yaml file that will configure kubernetes collector. This config should contain collector.kubernetes.all.events field with specified filter. Following example config will watch for normal events in default namespace (see Kubernetes Configuration):
Copy
collector:
  kubernetes:
    all:
      incluster: true
      kubeconfigpath: ""
      interval: "60s"
      events: |
        # Embedded YAML (as a multiline string literal)
        filters:
        - namespace: default
          type: normal
load:
  plugin: snap-plugin-collector-aokubernetes
  task: task-aokubernetes.yaml
  • If you want to monitor events count in AppOptics, then edit your current task-aokubernetes.yaml task manifest so it contains /kubernetes/events/count metric in workflow.collect.metrics list, and copy it to working directory:
Copy
version: 1
schedule:
  type: streaming
workflow:
  collect:
    config:
      /kubernetes:
        MaxCollectDuration: "2s"
        MaxMetricsBuffer: 250
    metrics:
      /kubernetes/events/count: {}
      /kubernetes/pod/*/*/*/status/phase/Running: {}
    publish:
    - plugin_name: publisher-appoptics
      config:
        period: 60
        floor_seconds: 60
  • Create logs.yaml file configuring the logs collector. Make sure that logs collector looks for /var/log/SolarWinds/Snap/events.log file:
Copy
collector:
  logs:
    all:
      loggly_token: <your loggly token>
      api_host: "logs-01.loggly.com"
      api_port: 514
      api_protocol: "tcp"
      connect_timeout: "30s"
      write_timeout: "30s"
      files: |
        /var/log/SolarWinds/Snap/events.log
      exclude_patterns: |
        .*self-skip-logs-collector.*
load:
  plugin: snap-plugin-collector-logs
  task: task-logs.yaml
  • Copy your current task-logs.yaml task manifest to working directory.
  • Once all 4 files are ready (kubernetes.yaml, logs.yaml, task-aokubernetes.yaml and task-logs.yaml), create 2 configmaps:
Copy
kubectl create configmap plugin-configs --from-file=./logs.yaml --from-file=./kubernetes.yaml --namespace=kube-system
kubectl create configmap task-manifests --from-file=./task-logs.yaml --from-file=./task-aokubernetes.yaml --namespace=kube-system
kubectl describe configmaps -n kube-system plugin-configs task-manifests
  • Edit swisnap-agent-deployment-event-collector.yaml and insert your APPOPTICS_TOKEN.
  • Create ServiceAccount and Deployment:
Copy
kubectl apply -f swisnap-agent-serviceaccount.yaml
kubectl apply -f swisnap-agent-deployment.yaml

Watch your cluster events in Loggly :)

Dashboard

After you've successfully set up the SolarWinds Snap Agent, enable the Kubernetes plugin in the AppOptics UI and you should see data pour in. More about Charts and Dashboards.

../../../../../../_images/kubernetes_dashboard.png

When the APM Integrated Experience is enabled, AppOptics shares a common navigation and settings with the other integrated experiences' products. How you navigate AppOptics and access its features may vary from these instructions. For more information, go to the APM Integrated Experience documentation.

The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.