Documentation forServer & Application Monitor
Monitoring your applications and environment is a key capability of SolarWinds Observability Self-Hosted (formerly Hybrid Cloud Observability) and is available in the Essentials edition. Server & Application Monitor (SAM) is also available in a standalone module.

Windows Network Load Balancing

This SAM application monitor template assesses the status and overall performance of Microsoft Windows Network Load Balancing by retrieving information from the MicrosoftNLB namespace and the Windows System Event Log.

Prerequisites

WMI access to the target server.

Credentials

Windows Administrator on the target server.

All Windows Event Log monitors should return zero values. A returned value other than zero indicates an abnormality. Examining the Windows system log files should provide information pertaining to the issue.

Component monitors

NLB Cluster Node status

This component monitor returns the current state of an Network Load Balancing (NLB) node.

Possible values:

  • 0 – Node is remote. The StatusCode value cannot be retrieved on the remote node
  • 1005 – Stopped: Cluster operations stopped on the node
  • 1006 – Converging: The cluster node is converging. Convergence is the process of redistributing the existing connection load to operational cluster nodes according to the current load balancing rules
  • 1008 – Converged: The cluster node converged successfully
  • 1009 – Draining: The cluster nodes are draining; meaning, this is a state in which a node is no longer accepting incoming traffic and is draining. No new connections are allowed, but existing connections are allowed to complete their jobs and terminate naturally. While draining, a node can participate in convergence and remains part of the cluster
  • 1013 – Suspended: Cluster operations are suspended on the node

Before using this monitor, set the correct NLB node name. Replace NLB_node with the NLB node name on which you applied this template in the WQL query section. If you assign the template on NLB_node1 and put NLB_node2 in the WQL query, the returned value will be zero.

Network Adapter Functionality

This monitor returns the number of events that occur when:

  • The NLB driver failed to bind or attach to the adapter;
  • The NLB failed to add a multicast MAC address to the network adapter;
  • The adapter does not support dynamic changing of its MAC address;
  • The NLB driver failed to register with the NDIS;
  • The NLB failed to update the adapter multicast list;
  • The MTU reported by the adapter is too small.

Type of event: Error. Event ID: 9, 50, 53, 85, 89, 90, 94, 98.

If you have problems with binding to the adapter, ensure that NLB is bound to an Ethernet network adapter.

If you have problems with MAC addresses, change the network adapter operating mode.

If the NLB driver failed to register or update the adapter multicast list, disable and re-enable NLB network adapters.

When the MTU is small, ensure that the MTU is properly configured.

NLB Bi-Directional Affinity (BDA) Configuration

This monitor returns the number of events that occur when:

  • An inconsistent teaming configuration is detected;
  • An invalid bi-directional affinity (BDA) team ID is detected;
  • An invalid BDA teaming port rule is detected;
  • The BDA team which this cluster attempted to join already has a designated master;
  • The BDA team in which this cluster participates, has no designated master;
  • This cluster has left a BDA team in which it was the designated master;
  • NLB failed to initialize BDA teaming on the adapter.

Type of event: Error and Warning. Event ID: 55, 56, 57, 59, 60, 62, 114.

Reconfigure the BDA teaming configuration. The BDA configuration must be identical on all cluster hosts. The team in which this cluster participates will be marked inactive and this cluster will remain in the converging state until consistent teaming configuration is achieved. You should first reconfigure the BDA configuration, and then restart the NLB cluster.

NLB Cluster Control

This monitor returns the number of events that occur when:

  • A version mismatch between the driver and control programs is detected;
  • The NLB driver failed to register the device object.

Type of event: Error. Event ID: 37, 88.

If a host is not running the same version of all NLB components as other hosts in the cluster, you should first delete the host that is not running the correct NLB version, remove NLB from the host, reinstall NLB, and then rejoin the cluster.

If the NLB driver fails to register a device, such as a network adapter, the cluster will converge and operate normally, but controlling the cluster might not work properly. You should disable all network adapters with NLB bound on this host, and then re-enable the adapters.

NLB Connection Tracking and Load Balancing

This monitor returns the number of events that occur when:

  • The NLB driver could not allocate enough memory resources to perform driver operations;
  • The maximum number of actively serviced connections that could be tracked by NLB is reached;
  • NLB cannot track TCP connections because it was unable to open the TCP connection callback object;
  • A load distribution error was detected during convergence;
  • NLB failed to register as a WMI provider;
  • The maximum number of actively serviced connections (using extended affinity) that could be tracked by NLB is reached.

Type of event: Error and Warning. Event ID: 10, 19, 81, 87, 115, 117.

If the NLB driver cannot allocate enough memory resources to operate the driver, you should close all programs on this cluster host that might be consuming memory, and then rebind NLB to the adapters. If this problem persists, you might need to add additional memory (RAM) to this host.

When the maximum number of actively serviced connections is reached, you can either add more hosts to the NLB cluster, (which distributes the number of incoming connections across more cluster hosts), or increase the connection tracking limit.

When NLB cannot track TCP connections or fails to register as a WMI provider, disable and re-enable NLB network adapters.

If load distribution errors were detected during convergence, this may result in either client traffic not being handled, general cluster traffic errors, or connections being reset. Convergence is a process by which hosts exchange messages to determine a new, consistent state of the cluster and to elect the default host. During convergence, a new load distribution is determined for hosts that share the handling of network traffic for specific TCP or UDP ports. To resolve the load distribution error, you should restart the NLB cluster.

NLB Dedicated IP (DIP) Addresses Functionality

This monitor returns the number of events that occur when:

  • The dedicated IP (DIP) address or mask is invalid;
  • NLB detected an unequal number of DIP addresses and network masks;
  • Duplicate DIP addresses were detected on the network;
  • NLB failed to add all the DIP addresses to this host because the maximum number of DIPs that can be added to this network adapter have been exhausted.

Type of event: Error. Event ID: 15, 30, 32, 83, 107.

You should verify that the dedicated IP address and subnet mask are correctly specified.

On all NLB cluster hosts, the dedicated IP addresses must have an equal number of subnet masks specified. If there are an unequal number, the NLB cluster will continue to operate, but the IP address that has no corresponding network mask will be ignored. To use this IP address, make sure that the number of IP addresses and network masks are the same.

All dedicated IP addresses must be unique in a NLB cluster.

If the number of dedicated IP addresses added to a network adapter has exceeded the maximum number allowed by the adapter, you will need to remove one or more IP addresses. The extra dedicated IP addresses will be ignored by the NLB cluster.

NLB Denial-of-service Protection

This monitor returns the number of events that occur when:

  • A SYN attack has been detected;
  • The NLB driver failed to open the SYN attack callback object;
  • The NLB driver failed to open the timer starvation callback object;
  • Timer starvation has been detected due to a denial of service attack or a very high server load.

Type of event: Error and Warning. Event ID: 92, 99, 104, 105.

Analyze the threats against the NLB cluster, including potential denial-of-service attacks, and then take the appropriate measures. If this is not an attack, the NLB cluster may be overloaded. To distribute the cluster traffic load over more hosts, you can add more hosts to the NLB cluster.

Disable and re-enable NLB network adapters.

NLB Extended Affinity Configuration

This monitor returns the number of events that occur when:

  • The NLB driver detects an inconsistency in the extended affinity configuration on the cluster host;
  • The NLB driver detects an inconsistency in the extended affinity configuration between cluster hosts.

Type of event: Warning. Event ID: 118, 119.

Confirm that the extended affinity configurations for all port rules are identical on all NLB hosts.

NLB Network Host Configuration

This monitor returns the number of events that occur when:

  • The NLB driver failed to initialize because the cluster IP, network address, or mask is invalid;
  • NLB detected duplicate cluster subnets;
  • The NLB cluster IGMP multicast IP address is invalid;
  • The NLB driver failed to register for notifications with the IPv4 or IPv6 NSI provider;
  • The virtual IP (VIP) address or mask is invalid;
  • NLB detected an unequal number of virtual IP (VIP) addresses and network masks.

Type of event: Error and Warning. Event ID: 14, 16, 18, 31, 73, 102, 103, 108, 109, 110, 112.

If the network media access control (MAC) address is not in the following format: XX-XX-XX-XX-XX-XX, where X is a hexadecimal value, it needs to be reconfigured.

If the NLB driver fails to initialize because the cluster IP address is not in a valid format, you should check that the network IP address is specified in a valid IPv4 or IPv6 address format.

If the NLB detects that there are duplicate subnets in the cluster, it may be due to network partitioning, which prevents NLB heartbeats of one or more hosts from reaching the other cluster hosts. You may need to restart the NLB cluster to resolve this issue.

If the NLB driver fails to initialize because the cluster network mask is not in a valid format, you should check that the network mask is specified in a valid format.

If the NLB cluster detects that the Internet Group Management Protocol (IGMP) multicast IP address is invalid, you should check the NLB configuration and make sure that the cluster IGMP multicast IP address is in a valid format.

If the NLB driver failed to register for notifications, the correct IP stack version (IPv4 or IPv6) must be installed on the network adapter to which NLB is bound. The virtual IP address must be in a valid IPv4 or IPv6 format.

The virtual IP address and mask must be in a valid IPv4 or IPv6 format. On all Network Load Balancing (NLB) cluster hosts, the virtual IP addresses must have an equal number of subnet masks specified.

NLB Host State Persistence

This monitor returns the number of events that occur when NLB failed to update the NLB host state in the registry

Type of event: Warning. Event ID: 74.

To check the initial NLB host state, you must first delete the registry key defined in the event log, and then confirm that the initial host state is correct.

Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on your computer.

NLB Port Rules Configuration

This monitor returns the number of events that occur when:

  • NLB failed to converge due to port rules with a duplicate single host priority in the cluster;
  • NLB failed to converge due to inconsistencies in the port rules between hosts;
  • Configured port rules conflict with the port rules of another host;
  • A port rule operation on the port was issued but there is no port rule that contains this port;
  • The NLB driver has detected one or more sessions corresponding to a port rule that is improperly configured;
  • The virtual IP (VIP) address in a port rule is invalid;

Type of event: Error and Warning. Event ID: 20, 21, 22, 25, 95, 111.

When single host filtering mode is used, traffic to the port or ports governed by that port rule is handled exclusively by the host whose priority has the lowest numeric value. When the host's single host priority is identical to the single host priority of another host, the cluster will not converge until the problem is corrected. You should check the NLB configuration of all port rules and make sure that each has a unique host priority (a number between 1 and 32).

When a NLB host in the cluster either contains a different number of port rules from another host, or its configured port rules conflict with the port rules of another host, the cluster will not converge until the problem is corrected. You should first ensure that all NLB hosts have identical port rules, and then, if there are port rules that are not identical and if there are not the same number of port rules on each NLB host, you should reconfigure the port rules to make them identical.

If there is no port rule that contains a specified port, you should confirm that the port rules are identical on all NLB hosts.

If the virtual IP address for a port rule is not in a valid format, the Network Load Balancing (NLB) cluster will converge and operate normally, but the port rule will be ignored. You should check that the virtual IP address is specified in a valid IPv4 or IPv6 address format.

NLB Host Configuration

This monitor returns the number of events that occur when:

  • NLB detected a duplicate host priority that is shared between cluster hosts;
  • NLB failed to query parameters from the registry key;
  • NLB failed to verify its parameters due to an improper configuration;
  • Host converged with legacy host(s) during rolling upgrades;
  • NLB received a heartbeat from a host with an invalid ID;
  • An unsupported legacy host was discovered on the network.

Type of event: Error and Warning. Event ID: 17, 34, 35, 86, 91, 97.

If a NLB host has a host priority that is identical to the host priority on another host, or the host priority is not valid, the cluster will not converge until the problem is corrected. The host priority must be a number from 1 through 32, and this value must be unique for all hosts in the cluster.

If a NLB host cannot process its configuration settings, you should confirm that the settings are correctly configured, and then, if changes are made, restart the NLB cluster.

A NLB cluster operating in a mixed mode (where hosts have different versions of an operating system installed) is only supported during rolling upgrades. Until all hosts are upgraded to the latest operating system version, newer NLB features will not be available. You should upgrade all hosts to the latest operating system version.

If an unsupported legacy host is discovered on the NLB cluster, you should remove the legacy host from the cluster. The cluster will remain in a converging state until all deprecated legacy hosts are removed.