3.1.0 - November 27, 2023

Return to top

New Features and Improvements:

• Prometheus is no longer required for the SWO K8s Collector.
• Existing bundled Prometheus deployment (configured with prometheus.enabled) will be automatically removed.
• The otel.metrics.extra_scrape_metrics option is deprecated and will be removed in future versions.
• Node metrics are now scraped from the node-collector daemonset for improved scalability.
• Application metrics from pods are now collected when annotated with prometheus.io/scrape=true.
• Network monitoring option added (disabled by default). Enable with ebpfNetworkMonitoring.enabled: true in values.yaml.
• Based on opentelemetry-network.
• For scaling configurations, refer to numIngestShards, numMatchingShards, and numAggregationShards in values.yaml.
• See exported_metrics.md for a list of metrics.
• Additional configuration parameters now available in values.yaml, such as sending_queue and retry_on_failure.

Breaking changes

• The otel.metrics.prometheus.url setting remains available, but only in conjunction with otel.metrics.extra_scrape_metrics.
• If otel.metrics.autodiscovery.prometheusEndpoints.enabled is set to true (default setting), the extra_scrape_metrics will be ignored to prevent duplicate collection. Override this by setting force_extra_scrape_metrics to true.

Fixes:

• Parameters reconfigured for enhanced performance. Message batch size has been adjusted to align with SWO limits.

Support Changes:

• SWO Agent image has been upgraded to v2.6.28.

2.8.0 - October 30, 2023

Return to top

New features and improvements

• Logs for Windows nodes are monitored.
• Services metrics are added (Scrape required kube_service_* and kube_endpoint_* metrics).

Fixes

• NodeSelector for kube-state-metrics is deployed on linux nodes only.
• Node name is detected for Fargate Nodes's metrics.
• Autofix from corrupted storage is added.

Support changes

• Log message attributes now respect OTEL log format.
• Updated labels so that resources can be identified more easily
• app.kubernetes.io/name is now the container application name (for example, swo-k8s-collector for "SWO k8s collector", swo-agent for "SWO agent").
• app.kubernetes.io/part-of is now always set to swo-k8s-collector.
• Logs daemon requests/limits are updated.
• Attributes net.host.name, net.host.port, http.scheme, prometheus, prometheus_replica and endpoint are removed from exported metrics.

2.7.0 - September 4, 2023

Return to top

New features and improvements

• Added new Helm settings aws_fargate.enabled and aws_fargate.logs.* that allow the SWO K8s Collector Helm chart to setup AWS EKS Fargate logging ConfigMap.
• Setting prometheus.forceNamespace can be used to force deployment of the bundled Prometheus to a specific non-Fargate namespace.
• Updating k8sattributes to instrument attribute indicating that object exists.
• Metrics no longer send k8s.node.name resource attribute if the node does not exists in Kubernetes (for example: Fargate nodes).
• Events collectiondoes not produce resource attributes for entities that do not exists in Kubernetes.
• DaemonSet for Log collection now restricts where it runs so that Fargate nodes are excluded and only linux nodes with amd64 architecture are included.

Fixes

• Journal logs are collected correctly on EKS and other environment where journal logs are stored in /var/log/journal.

Support changes

• Update library/golang from 1.20.7-bullseye to 1.21.0-bullseye and update some 3rd party dependencies.
• OTEL upgraded to v0.81.0.

2.6.0 - July 18, 2023

Return to top

New features and improvements

• k8s.pod.spec.cpu.limit is calculated from kube-state-metrics (Kubernetes describe) and not from container runtime metrics. This should make the metric more reliable.
• Original Prometheus metrics are not modified.

2.5.0 - June 12, 2023

Return to top

New features and improvements

• Scrapes storage metrics (Persistent Volume and Persistent Volume Claim).

Fixes

• CPU/Memory usage Pod metrics are fixed on latest cAdvisor/containerd.
• Node network metrics are fixed for environments using Docker runtime (and other runtimes where Pod network metrics are not available).

2.4.0 - May 16, 2023

Return to top

New features

• New container metrics k8s.container.fs.iops, k8s.container.fs.throughput, k8s.container.network.bytes_received, and k8s.container.network.bytes_transmitted are added.
• kube_pod_init_container_* metrics are scraped.
• Metrics k8s.container.spec.cpu.limit, k8s.container.spec.cpu.requests, k8s.container.spec.memory.requests, k8s.container.spec.memory.limit, and k8s.container.status now include datapoints for both init and non-init containers.
• kube-state-metrics is now bundled with the Helm chart so that its metrics are predictable.
• FIPS compliance is added.

Breaking changes

• Metrics k8s.cluster.memory.utilization and k8s.cluster.cpu.utilization have been removed and replaced by composite metrics calculated by SolarWinds Observability.

Fixes

• Updates to build dependencies to improve security.
• The auto-update feature is fixed.
• Permissions are adjusted to be able to update ClusterRoles for future increments.
• The update is now atomic. If the update fails for any reason, it will rollback rather than leave Helm release in a Failed state.
• Metric k8s.kube_pod_status_phase should not send values with 0 anymore.

2.3.0 - April 13, 2023

Return to top

New features

• Deploys Prometheus as part of the SWO K8s Collector Helm chart installation, controlled by setting prometheus.enabled: true in values.yaml.
• Automatically extracts Kubernetes labels and annotations from K8s resources, such as Pods, Namespaces, Deployments, StatefulSets, ReplicaSets, DaemonSets, Jobs, CronJobs, and Nodes. Labels and annotations are sent as resource attributes with metrics end events, and can be used to filter metrics in the Metrics Explorer, Alerts, and custom dashboards.
• Monitors a new StatefulSet with a light weight version of the SolarWinds Observability Agent optionally deployed by default, which allows integrations to be attached to it.
• Adds syslog attributes to log entries so logs are properly displayed in the Logs Explorer (syslog.facility, syslog.version, syslog.procid, syslog.msgid).
• Adds resource level attributes: host.hostname to entities, which contains the name of the pod (represented as System in the Logs Explorer), and service.name, which contains the name of the container (represented as Program in the Logs Explorer).
• New metrics are scraped from Prometheus: k8s.kube_replicaset_spec_replicas, k8s.kube_replicaset_status_ready_replicas, k8s.kube_replicaset_status_replicas
• The metric k8s.cluster.version is sent to SolarWinds Observability, which extracts the version from kubernetes_build_info. The metric kubernetes_build_info is no longer published.

Fixes

• The option honor_labels is enabled to keep scraped labels unchanged.
• The k8s.job.condition resource attribute is fixed to properly handle a Failed state.
• The calculation of k8s.pod.spec.memory.limit on a newer container runtime is fixed to use kube_pod_container_resource_limits instead of container_spec_memory_limit_bytes.
• Grouping conditions for container_network_* and container_fs_* metrics do not rely on the container attribute.

2.2.0 - March 23, 2023

Return to top

New features

• Auto-updating the SWO K8s Collector is available with the optional autoupdate.enabled setting in values.yaml.
• Kubernetes Jobs are now collected.

Improvements

• Events instrumentation is improved with a new k8s.pod.name attribute added to events.
• Metrics instrumentation is improved with:
  • a new internal IP attribute for the node.
  • a new k8s.kubernetes_build_info metric, which includes the cluster version.
  • a new k8s.container.spec.cpu.limit metric, which includes the container's CPU quota.
  • a new k8s.container.cpu.usage.seconds.rate metric, which includes the container CPU usage.
  • new container.id and container.runtime attributes added to the to k8s.kube_pod_container_info metric for unique container identification.
  • scraping of Init containers.
• PodMonitor resources can be deployed, allowing OTEL Collector telemetry to be scraped by the Prometheus Operator (see Prometheus Operator design).

Fixes

• Internal metrics generated by the Prometheus scraper are no longer sent, decreasing traffic (scrape_duration_seconds, scrape_samples_post_metric_relabeling, scrape_samples_scraped, scrape_series_added, up)
• Datapoints for internal k8s containers (with the name "POD", usually using image "pause") are filtered out. Filtering out these datapoints prevents creating extra container entities.

Support changes

• OTEL Collector is upgraded to 0.73.

2.1.0 - February 16, 2023

Return to top

• If collector telemetry is enabled (true by default), expose it on container port 8888. The OTel collectors also contain Prometheus annotations so that telemetry is discovered and collected by Prometheus within the monitored cluster.
• Add configurable log collector pod tolerations that allow it to be deployed on tainted nodes so that we can also collect logs from those nodes.
• Configuration of file_storage extension is now available in values.yaml.
• Configuration of filelogreciever is now available in values.yaml.
• A new events transform pipeline is added, which sets sw.namespace attribute to sw.events.inframon.k8s.

Improvements

• OTel collector image uses 0.3.0. The OTel collector image 0.3.0 updates OTel to 0.69.0 and adds file_storage so it can be in processors.

Fixes

• Properly annotate configmap's checksums.
• Fix previously ignored https_proxy_url configuration.
• Log collector has optimizations to prevent Out of Memory situations.
  • Checkpoints persisted in persistent storage using file_storage extension, rather than in memory.
  • Memory optimization to reduce amount of concurrent log scans by setting max_concurrent_files to 10 (from the default of 1024).
  • Default memory limit is 700MiB (which should be enough for large logs).
  • Difference between the OTel memory limit and K8s memory limit is now a default value of 150MiB, to give OTel enough buffer to avoid running out of memory and crashing.

2.0.2 - January 18, 2023

Return to top

New features

• Initial Helm release.
• Create Error reason as mapping from combination of Event Reason and Type fields.
• Add support for HTTPS proxies.