Documentation forServer & Application Monitor

Best practices for SAM templates, application monitors, and script monitors

Creating SAM templates and application monitors involves more than just adding and configuring component monitors. Use these best practices, tips, and tricks about performance enhancements, testing, and scripting to create and customize templates and component monitors.

In addition to these tips, note the following details about SAM templates:

  • SolarWinds recommends checking THWACK periodically for updates to predefined SAM templates. Except for AppInsight templates, templates are not updated automatically during upgrades to avoid overwriting custom changes made to templates. For details, see Import and export SAM templates.
  • Periodically, SolarWinds releases SAM templates to support the latest product versions such as Microsoft Server 2016. You can continue using templates for older product versions, but updating to the latest template is recommended.

Performance enhancements

Starting in SAM 2020.2, you can improve scalability and performance by enabling WinRM as the primary polling method for WMI-based component monitors. See Best practices for application monitor polling in SAM.

Modify the polling frequency for performance

Depending on the length of calls and amount of data pulled for a monitor, you may want to modify the frequency. For script monitors you may need to only run the script once per day or once per week. For example, to compare MIBs using the SolarWinds MIB Database template, you may only need to run the comparison once a day or week.

Extend the polling timeout for long calls

For scripts with lengthy calls for large amounts of data, extend the polling timeout. The default 300 seconds may not be long enough to process scripts. If the call may take longer, especially during peak times, increase the timeout. For example, for MIB database comparison scripts using the SolarWinds MIB Database template, multiple files are called, downloaded, and compared to return status messages and complete specific actions.

Enhance latency and performance by pulling multiple metrics per template

When executing script component monitors in a template, SAM affects performance and latency making calls to a target server. Complete calls for up to 10 metrics per script to reduce the number of calls, increasing performance. Depending on the size and processing of scripts, balance scripts and lengthy calls across multiple instances of a script monitor.

Script, monitor, and template testing

As detailed in the SAM Custom Template Guide, script component monitors (also called "script monitors") offer limitless options for monitoring and returning metrics for target servers. Each monitor has different options to execute scripts, add credentials, set working directories, and then generate and display returned values as output. Test the script component monitor after adding a script to properly calibrate the monitor, generate tables, and verify correct communication between the target node, the Orion server, and the template.

SAM's Component Monitor Library includes the following predefined script monitors, but you can also write your own.

Check credentials and server permissions for scripts

Verify you have the correct credentials with assigned account permissions to execute scripts on the Orion server and target server. The script monitors may provide fields for credentials, or you may need to provide credentials in the script code, arguments, or command line. Test the script in SAM prior to verifying credentials and access.

Test scripts before monitoring

When adding and configuring script component monitors, test scripts. When the test completes, SAM registers each returned metric as a numbered output in the Orion database. You can configure the display of collected metrics and values through the component monitor. Each script monitor supports up to 10 different outputs.

Confirm node status updates

Until tested, scripts and component monitors return an initial "Unknown" status. After testing, polling returns accurate application status.

Disclaimer: Scripts provided outside of the Orion Web Console are not supported under any SolarWinds support program or service. Scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.

Script best practices

Use code comments

Code comments help document the intent for code, decisions made, and to track changes. SolarWinds recommends using code comments to keep detailed steps and responses in your code. If additional administrators work in the script monitors, the comments provide context for the code.

# for a comment per line.
For lengthy comments per code section.

Do not use positional parametrization

In the command line for executing scripts, always add the parameter per value. Do not assume the position of data in the command dictates the parameter. For example, use -h for hostname.

Use a header for writing multiple scripts

Create a header in your code to reuse throughout your scripts. The header could include example code and code comments for:

  • A listed of exit codes.
  • Set variables for return metrics commonly used in your scripts.
  • Use code to determine if you are testing code on the Orion server or target server. For example, the following PowerShell code returns a message identifying if the server is a test system or the Orion server:

    Additionally, you could add a step to save the code to the Orion server if it's not already there.

Use SolarWinds macros

When using SolarWinds macros, consider assigning them to named variables in your scripts.

The following macros are available for Linux/Unix, Nagios, Windows Script, and PowerShell script monitors:

  • ${USER}
  • ${PORT}
  • ${Node.SysName}
  • ${Node.Caption}
  • ${Node.DNS} - Use this instead of ${IP}.
  • ${Node.ID}
  • ${Component.ID}
  • ${Component.Name}
  • ${Application.Id}
  • ${Application.Name}
  • ${Application.TemplateId
  • ${Threshold.Warning}
  • ${Threshold.Critical}
  • Node Custom Property Macros ${Node.CustomPropertyName}
  • Application Custom Property Macros ${Application.CustomPropertyName}

For agent monitored nodes, use the macros ${Node.SysName} and ${Node.DNS}. The ${IP} may return a loopback IP before polling starts.

Report status through exit codes

Scripts must report their status by exiting with the appropriate exit code. The exit code is used to report the status of the monitor, as displayed in the Orion Web Console.

A script should return an exit code which results in an Up (0), Warning (2), or Critical (3) status. When an exit code is received, the appropriate dynamic evidence table structure is created and all further exit codes are handled correctly. If the component only returns Down (1) or Unknown (4) on first use, the appropriate dynamic evidence table structure is not created appropriately.

  • 0 - Up
  • 1 - Down
  • 2 - Warning
  • 3 - Critical
  • Any other value - Unknown, for example 4

Multiple options for returning exit code and message

You can return one of multiple options for exit codes and messages using IF/ELSE or case statements in scripts.

Use error trapping to capture issues

Using error trapping code such as try/catch blocks help capture and report errors, thus blocks providing more details about an issue.

Your organization should internally review and assess to what extent PowerShell is incorporated into your environment. This is especially important when importing scripts from third parties, including content posted by other customers in the SolarWinds online IT community, THWACK. For details, see Use PowerShell in SAM.