4.3.0 - December 18, 2024

Return to top

New features and improvements

• Added observation of changes to ConfigMaps.

• Helm settings for the deployed SWO K8s Collector are now included in the data sent to SolarWinds Observability.

• Added initial support for auto instrumentation of Kubernetes Containers with SolarWinds Observability APM using the OpenTelemetry Operator.

• Removed name and version library attributes from the OTel scope context.

• Added support for metrics discovery on pods running on AWS Fargate nodes.

• The storage class for the bundled SolarWinds Observability Agent is now configurable.

Breaking changes

• Metrics, logs, and events are no longer annotated with labels and annotations of related Kubernetes resources. The previously deprecated settings k8s_instrumentation are now ignored.

  Labels and annotations for supported Kubernetes resources are still collected with resource manifests.

Fixes

• Fixed collecting logs from Windows Nodes based on new AWS EKS AMIs.

Support changes

• Upgraded the OTel collector image to 0.12.1. This brings the following changes:

  • Bumped 3rd party dependencies and Docker images.

  • Bumped Go builder image to 1.23.4.

  • Upgraded OTel Collector to v0.113.0.

  • For details, see the release notes for version 0.12.1.

• Added init containers performing the SolarWinds Observability OTel endpoint connection check to all SWO K8s Collector pods.

• Bumped the version of the bundled SolarWinds Observability Agent.

• Increased default memory usage limits for Kubernetes events collection.

4.2.0 - November 13, 2024

Return to top

New features and improvements

• Added collection of metrics for cert-manager, etcd, controller-manager, apiserver and coredns services.

• Added collection of additional Kubernetes Volume metrics.

• Removed dependency on grpcurl from the SolarWinds Observability SaaS OTel endpoint connection check.

• Added support for HTTPS endpoints in the metrics discovery.

• Added setting otel.metrics.autodiscovery.prefix for configuring prefix of discovered metrics (k8s. by default).

Support changes

• Upgraded the OTel collector image to 0.11.8. This brings the following changes:

  • Bumped 3rd party dependencies and Docker images.

  • All images (including Windows) are now published as one multiarch image.

  • For details, see the release notes for version 0.11.8.

• Replaced OTel Network Collector images with a new version. This brings the following changes:

  • The images are now stored in DockerHub repository solarwinds.

  • Bumped 3rd party dependencies.

  • Added support for ARM Kubernetes Nodes.

• Bumped the version of the bundled SolarWinds Observability Agent.

• Upgraded the kube-state-metrics subchart to 5.26.0.

4.1.0 - September 30, 2024

Return to top

New features and improvements

• Improved compatibility with 3rd party tools for managing Helm charts:

  • Added Helm chart README.

  • Fixed icon.

  • Improved Helm chart documentation.

• Setting the cluster.uid value is now optional. If not specified, the value defaults to the value of cluster.name.

• Added initial support for OpenShift clusters.

  The feature is part of an early access program. All feature functionality is subject to change.

  See Monitor an OpenShift cluster for more information.

Fixes

• Configuring otel.events.k8s_instrumentation.labels.excludePattern or otel.events.k8s_instrumentation.annotations.excludePattern might cause the events collector to fail when manifest collection is enabled.

• Fixed the autoupdate job to run only on Linux nodes.

Support changes

• Upgraded the OTel collector image to 0.11.5. This brings the following changes:

  • Bumped 3rd party dependencies and Docker images.

  • For details, see the release notes for version 0.11.5.

• Settings otel.metrics.k8s_instrumentation, otel.logs.k8s_instrumentation, and otel.events.k8s_instrumentation are deprecated and may be removed in a future version.

• Enabled automatic deployment of the OTel Network Collector to the cluster.

4.0.0 - August 28, 2024

New features and improvements

• Added observation of changes to supported resources and sending of their full manifests to SolarWinds Observability SaaS.

  • The equivalent of kubectl get <resource> <some-resource-name> -o=yaml --show-managed-fields.

  • The current list of watched resources: pods, deployments, statefulsets, replicasets, daemonsets, jobs, cronjobs, nodes, services, persistentvolumes, persistentvolumeclaims, ingresses, and istio virtualservices.

  • Enabled by default. This can be disabled by setting otel.manifests.enabled to false.

  • Currently runs in the event collector, so otel.events.enabled must be set to true (default).

• Improved filtering abilities.

Breaking changes

• Added validation of the OTel endpoint provided in values.yaml. Deprecated endpoints are now rejected, and an error is reported during chart installation or upgrade. Only endpoints in the format otel.collector.xx-yy.cloud.solarwinds.com are accepted. See Data centers and endpoint URIs for the list of supported endpoints.

• Removed deprecated prometheus, prometheus-node-exporter, and opencost sub-charts.

• Added collection of all container logs by default.

Fixes

• Adjusted the setting otel.https_proxy_url to also be applied to the SWO Agent and to disable SolarWinds Observability OTel endpoint check init container.

Support changes

• Changed the otel.metrics.filter, otel.logs.filter, and otel.events.filter settings to enable use of OTTL syntax and access resource attributes:

  • If you are using the old filtering syntax, they behave like in 3.x.x and previous versions of the k8s collector. If you switch to using the new syntax, some of the metric/log-level attributes, such as k8s.deployment.name, become resource attributes.

• Removed custom swmetricstransform processor (replaced with filter).

• Switched to using container parser in the filelog receiver for parsing container logs and their metadata.

• Upgraded collector image to 0.11.4, which includes the following changes:

  • Removed deprecated ballastextension. The collector now uses environment variable GOMEMLIMIT.

  • Bumped Go builder image to 1.22.6.

  • Bumped third-party dependencies and Docker images.

  • Upgraded OTel Collector to v0.107.0.

• Upgraded SWO Agent image to v2.9.3.

• Upgraded grpcurl image to v1.9.1.

• Upgraded the kube-state-metrics subchart from 5.15.2 to 5.25.1, which included the following changes:

  • Upgraded the kube-state-metrics image to 2.13.0.

  • Added restart_policy to the kube_pod_init_container_info metric.

  • Added new metric kube_pod_container_status_last_terminated_timestamp.

3.4.2 - August 20, 2024

Return to top

Fixes

• Improved detection of Kubernetes cluster version.

Support changes

• Upgraded the OTel collector image to 0.10.3. This brings the following changes:

  • Bumped 3rd party dependencies and Docker images.

  • For details, see the release notes for version 0.10.3.

• Bumped the version of the bundled SolarWinds Observability Agent to 2.8.90.

Important notes

• The bundled OpenCost deployment has been deprecated and may be removed in the next major version. If you are using OpenCost by setting opencost.enabled: true in your values.yaml file, consider deploying your own solution.

• The next major version will no longer accept previously deprecated SolarWinds Observability SaaS OTel endpoints. Only endpoints in the format otel.collector.xx-yy.cloud.solarwinds.com will be accepted. See Datacenters and endpoint URIs for the list of supported endpoints.

3.3.0 - April 26, 2024

Return to top

New features and improvements

• Added instrumentation of workload attributes to collected logs (k8s.deployment.name, etc.). The instrumentation of labels and annotations is disabled by default.

• Added option to configure nodeSelector and affinity for the SolarWinds Observability Agent.

• Added option to configure the timeout for each attempt to send data to SolarWinds Observability SaaS.

  The default is 15s (previously it was 5s) to avoid unnecessary retries when the endpoint takes its time to respond.

Fixes

• Fixed Journal log collection on AWS EKS (and other environments where journal logs are stored in /var/log/journal).

Support changes

• Upgraded the OTel collector image to 0.10.0. This brings the following changes:

  • Dropped support for several Windows versions that are out of support. The minimum requirement now is Windows 10 or Windows Server 2016.

  • Bumped 3rd party dependencies and Docker images.

  • OTel upgraded to 0.98.0.

  • For details, see the release notes for version 0.10.0.

• Added validation schema for the provided Helm chart configuration.

  If an incorrect configuration is provided, the installation/update of a Helm release will end in error. Previous versions silently ignored the incorrect parts.

• Container logs from AWS EKS Fargate clusters are now sent to SolarWinds Observability SaaS as-is. fluentbit.io/parser and fluentbit.io/exclude annotations are ignored. This both fixes an issue with "empty" JSON logs received for Fargate containers and aligns the behavior with non-Fargate container logs.

  This change is only applied to Pods started after the new SWO K8s Collector is deployed to the k8s cluster.

• Added validation of the OTel endpoint provided in values.yaml. In case a deprecated endpoint is detected, a warning is shown during chart installation/update.

3.2.0 - February 5, 2024

Return to top

New features and improvements

• Added support for Linux ARM64.

• Added support for Windows Server 2019.

• Added option to deploy the OpenCost collector.

  The collector can be enabled by setting the openconst.enabled value to true in values.yaml.

• Added the option to pull images from ACR by setting the global.azure.images.<image_key> value in values.yaml when running in AKS.

• Added the otel.api_token value to allow setting the API token for the OTel collector through values.yaml.

• Added the otel.metrics.autodiscovery.prometheusEndpoints.podMonitors configuration option. You can define the option if you want to monitor applications that do not have Prometheus annotations.

• Started publishing custom Istio metrics when available: k8s.istio_request_bytes.delta, k8s.istio_response_bytes.delta, k8s.istio_requests.rate, k8s.istio_tcp_sent_bytes.rate, k8s.istio_tcp_received_bytes.rate, k8s.istio_requests.delta, k8s.istio_tcp_sent_bytes.delta, k8s.istio_tcp_received_bytes.delta, k8s.istio_request_bytes.rate, k8s.istio_response_bytes.rate, and k8s.istio_request_duration_milliseconds.rate.

Fixes

• Autoupdate job is fixed.

• Usage of ebpfNetworkMonitoring.k8sCollector.relay.image.pullPolicy is fixed.

• Restarts of the collector when automatic discovery and scraping of Prometheus endpoints (otel.metrics.autodiscovery.prometheusEndpoints.enabled) was disabled or enabled on a Fargate environment were fixed.

Support changes

• Upgraded the OTel collector image to 0.9.2. This brings the following changes:

  • Switched from scratch base image to distroless.

  • Bumped 3rd party dependencies and Docker images.

  • OTel upgraded to 0.91.0.

  • For details, see the release notes for versions 0.8.11, 0.8.12, 0.8.13, 0.9.0, 0.9.1, 0.9.2.

• Adjusted node-collector memory limits for better performance.

• Removed sha256 from image references to enable multiarch support.

• Removed k8s.kube_daemonset_labels, k8s.kube_deployment_labels, and k8s.kube_statefulset_labels metrics. These are no longer published by the updated kube-state-metrics.

• Stopped sending histogram metrics to SolarWinds Observability SaaS.

• Added PVC for SWO Agent when enabled.

3.1.0 - November 27, 2023

Return to top

New Features and Improvements:

• Prometheus is no longer required for the SWO K8s Collector.
• Existing bundled Prometheus deployment (configured with prometheus.enabled) will be automatically removed.
• The otel.metrics.extra_scrape_metrics option is deprecated and will be removed in future versions.
• Node metrics are now scraped from the node-collector daemonset for improved scalability.
• Application metrics from pods are now collected when annotated with prometheus.io/scrape=true.
• Network monitoring option added (disabled by default). Enable with ebpfNetworkMonitoring.enabled: true in values.yaml.
• Based on opentelemetry-network.
• For scaling configurations, refer to numIngestShards, numMatchingShards, and numAggregationShards in values.yaml.
• See exported_metrics.md for a list of metrics.
• Additional configuration parameters now available in values.yaml, such as sending_queue and retry_on_failure.

Breaking changes

• The otel.metrics.prometheus.url setting remains available, but only in conjunction with otel.metrics.extra_scrape_metrics.
• If otel.metrics.autodiscovery.prometheusEndpoints.enabled is set to true (default setting), the extra_scrape_metrics will be ignored to prevent duplicate collection. Override this by setting force_extra_scrape_metrics to true.

Fixes:

• Parameters reconfigured for enhanced performance. Message batch size has been adjusted to align with SWO limits.

Support Changes:

• SWO Agent image has been upgraded to v2.6.28.

2.8.0 - October 30, 2023

Return to top

New features and improvements

• Logs for Windows nodes are monitored.
• Services metrics are added (Scrape required kube_service_* and kube_endpoint_* metrics).

Fixes

• NodeSelector for kube-state-metrics is deployed on linux nodes only.
• Node name is detected for Fargate Nodes's metrics.
• Autofix from corrupted storage is added.

Support changes

• Log message attributes now respect OTel log format.
• Updated labels so that resources can be identified more easily
• app.kubernetes.io/name is now the container application name (for example, swo-k8s-collector for "SWO k8s collector", swo-agent for "SWO agent").
• app.kubernetes.io/part-of is now always set to swo-k8s-collector.
• Logs daemon requests/limits are updated.
• Attributes net.host.name, net.host.port, http.scheme, prometheus, prometheus_replica and endpoint are removed from exported metrics.

2.7.0 - September 4, 2023

Return to top

New features and improvements

• Added new Helm settings aws_fargate.enabled and aws_fargate.logs.* that allow the SWO K8s Collector Helm chart to setup AWS EKS Fargate logging ConfigMap.
• Setting prometheus.forceNamespace can be used to force deployment of the bundled Prometheus to a specific non-Fargate namespace.
• Updating k8sattributes to instrument attribute indicating that object exists.
• Metrics no longer send k8s.node.name resource attribute if the node does not exist in Kubernetes (for example Fargate nodes).
• Events collection does not produce resource attributes for entities that do not exist in Kubernetes.
• DaemonSet for Log collection now restricts where it runs so that Fargate nodes are excluded and only Linux nodes with amd64 architecture are included.

Fixes

• Journal logs are collected correctly on EKS and other environments where journal logs are stored in /var/log/journal.

Support changes

• Update library/golang from 1.20.7-bullseye to 1.21.0-bullseye and update some 3rd party dependencies.
• OTel upgraded to v0.81.0.

2.6.0 - July 18, 2023

Return to top

New features and improvements

• k8s.pod.spec.cpu.limit is calculated from kube-state-metrics (Kubernetes describe) and not from container runtime metrics. This should make the metric more reliable.
• Original Prometheus metrics are not modified.

2.5.0 - June 12, 2023

Return to top

New features and improvements

• Scrapes storage metrics (Persistent Volume and Persistent Volume Claim).

Fixes

• CPU/Memory usage Pod metrics are fixed on the latest cAdvisor/containerd.
• Node network metrics are fixed for environments using Docker runtime (and other runtimes where Pod network metrics are not available).

2.4.0 - May 16, 2023

Return to top

New features

• New container metrics k8s.container.fs.iops, k8s.container.fs.throughput, k8s.container.network.bytes_received, and k8s.container.network.bytes_transmitted are added.
• kube_pod_init_container_* metrics are scraped.
• Metrics k8s.container.spec.cpu.limit, k8s.container.spec.cpu.requests, k8s.container.spec.memory.requests, k8s.container.spec.memory.limit, and k8s.container.status now include datapoints for both init and non-init containers.
• kube-state-metrics is now bundled with the Helm chart so that its metrics are predictable.
• FIPS compliance is added.

Breaking changes

• Metrics k8s.cluster.memory.utilization and k8s.cluster.cpu.utilization have been removed and replaced by composite metrics calculated by SolarWinds Observability SaaS.

Fixes

• Updates to build dependencies to improve security.
• The auto-update feature is fixed.
• Permissions are adjusted to be able to update ClusterRoles for future increments.
• The update is now atomic. If the update fails for any reason, it will rollback rather than leave Helm release in a Failed state.
• Metric k8s.kube_pod_status_phase should not send values with 0 anymore.

2.3.0 - April 13, 2023

Return to top

New features

• Deploys Prometheus as part of the SWO K8s Collector Helm chart installation, controlled by setting prometheus.enabled: true in values.yaml.
• Automatically extracts Kubernetes labels and annotations from K8s resources, such as Pods, Namespaces, Deployments, StatefulSets, ReplicaSets, DaemonSets, Jobs, CronJobs, and Nodes. Labels and annotations are sent as resource attributes with metrics end events, and can be used to filter metrics in the Metrics Explorer, Alerts, and custom dashboards.
• Monitors a new StatefulSet with a light weight version of the SolarWinds Observability Agent optionally deployed by default, which allows integrations to be attached to it.
• Adds syslog attributes to log entries so logs are properly displayed in the Logs Explorer (syslog.facility, syslog.version, syslog.procid, syslog.msgid).
• Adds resource level attributes: host.hostname to entities, which contains the name of the pod (represented as System in the Logs Explorer), and service.name, which contains the name of the container (represented as Program in the Logs Explorer).
• New metrics are scraped from Prometheus: k8s.kube_replicaset_spec_replicas, k8s.kube_replicaset_status_ready_replicas, k8s.kube_replicaset_status_replicas
• The metric k8s.cluster.version is sent to SolarWinds Observability SaaS, which extracts the version from kubernetes_build_info. The metric kubernetes_build_info is no longer published.

Fixes

• The option honor_labels is enabled to keep scraped labels unchanged.
• The k8s.job.condition resource attribute is fixed to properly handle a Failed state.
• The calculation of k8s.pod.spec.memory.limit on a newer container runtime is fixed to use kube_pod_container_resource_limits instead of container_spec_memory_limit_bytes.
• Grouping conditions for container_network_* and container_fs_* metrics do not rely on the container attribute.

2.2.0 - March 23, 2023

Return to top

New features

• Auto-updating the SWO K8s Collector is available with the optional autoupdate.enabled setting in values.yaml.
• Kubernetes Jobs are now collected.

Improvements

• Events instrumentation is improved with a new k8s.pod.name attribute added to events.
• Metrics instrumentation is improved with:
  • A new internal IP attribute for the node.
  • A new k8s.kubernetes_build_info metric, which includes the cluster version.
  • A new k8s.container.spec.cpu.limit metric, which includes the container's CPU quota.
  • A new k8s.container.cpu.usage.seconds.rate metric, which includes the container CPU usage.
  • New container.id and container.runtime attributes added to the to k8s.kube_pod_container_info metric for unique container identification.
  • Scraping of Init containers.
• PodMonitor resources can be deployed, allowing OTeL Collector telemetry to be scraped by the Prometheus Operator (see Prometheus Operator design).

Fixes

• Internal metrics generated by the Prometheus scraper are no longer sent, decreasing traffic (scrape_duration_seconds, scrape_samples_post_metric_relabeling, scrape_samples_scraped, scrape_series_added, up)
• Datapoints for internal k8s containers (with the name "POD", usually using image "pause") are filtered out. Filtering out these datapoints prevents creating extra container entities.

Support changes

• OTel Collector is upgraded to 0.73.

2.1.0 - February 16, 2023

Return to top

• If collector telemetry is enabled (true by default), expose it on container port 8888. The OTel collectors also contain Prometheus annotations so that telemetry is discovered and collected by Prometheus within the monitored cluster.
• Add configurable log collector pod tolerations that allow it to be deployed on tainted nodes so that we can also collect logs from those nodes.
• Configuration of file_storage extension is now available in values.yaml.
• Configuration of filelogreciever is now available in values.yaml.
• A new events transform pipeline is added, which sets sw.namespace attribute to sw.events.inframon.k8s.

Improvements

• OTel collector image uses 0.3.0. The OTel collector image 0.3.0 updates OTel to 0.69.0 and adds file_storage so it can be in processors.

Fixes

• Properly annotate configmap's checksums.
• Fix previously ignored https_proxy_url configuration.
• Log collector has optimizations to prevent Out of Memory situations.
  • Checkpoints persisted in persistent storage using file_storage extension, rather than in memory.
  • Memory optimization to reduce amount of concurrent log scans by setting max_concurrent_files to 10 (from the default of 1024).
  • Default memory limit is 700MiB (which should be enough for large logs).
  • Difference between the OTel memory limit and K8s memory limit is now a default value of 150MiB, to give OTel enough buffer to avoid running out of memory and crashing.

2.0.2 - January 18, 2023

Return to top

New features

• Initial Helm release.
• Create Error reason as mapping from combination of Event Reason and Type fields.
• Add support for HTTPS proxies.