Documentation forSolarWinds Observability

Logs Explorer

When entities are configured for SolarWinds Observability, logs may be detected or brought in alongside the other data. For example, if a host is monitored, logs on the host may be detected and sent to SolarWinds Observability. Logs can also be sent directly to SolarWinds Observability, whether or not they correspond to a monitored entity. See Collect logs.

All logs are shown in the Logs Explorer.

Open the Logs Explorer

In the left pane, click Logs. The Logs Explorer lists the logs received for all of your observed entities.

Search logs

Use the Search box to search for a specific log or to filter the logs. The Search field uses simple text search.

When you enter syntax into the Search box, the Logs Explorer displays only the logs that match the syntax.

Search term examples Explanation
status error Logs Explorer displays only events that contain "status" and "error" in any part of the message.
"status code" Logs Explorer displays only events that contain "status code".
"eventVersion":"1.08" Logs Explorer displays only events that contain an exact match to "eventVersion" followed by "1.08".

The search syntax is similar to Google search. For example, you can use phrases (in single or double quotes), logical operators (AND, OR), and exclusion (-).

For more information on search syntax, see Use advanced search syntax or Use JSON search syntax.

Perform time/date searches

Click the clock icon to the right of the search field to jump to logs received on a specific date and time in the searchable history.

In the Seek To field, enter a time, date, or combination. Use the International Organization for Standardization (ISO) date format: YYYY-MM-DDTHH:MM:SS.

For example: 2021-09-30T12:00:00.

The contents of Logs Explorer display logs received at the specified time.

Understand timezones as they relate to time/date searches

Display preferences affect the way time/date searches display results. For example, if user's display preference is set to Use UTC Timestamps, the timezone is always interpreted as UTC. If the display preference is not set, the following occurs.

  • When searching by time in the Logs Explorer, the searched-for timestamp is interpreted in the timezone set in the user profile.

  • SolarWinds Observability tracks the time a log message was received, and automatically displays it in the profile timezone. Users can have different timezones within the same organization, so there is less need to perform mental timezone conversions.

Save searches

Save searches for future use by clicking the folder icon inside the search bar, and then clicking Save Search. Provide a name, and optionally, select Private to make the saved search visible only to you.

Reuse saved searches by clicking the folder icon inside the search bar, and then selecting your saved search.

Create a quick alert based on a search

Create an alert based on a search by clicking the Alert button. Conditions of the current search will auto-populate to the new alert configuration. For more information, see Alerts.

View log item details

Click the purple plus (+) or minus (-) button within a log item to expand or contract it.

Pause/restart live log events

By default, Logs Explorer shows events as they happen. The Logs Explorer is updated all the time. Logs are live.

  • To pause the live tail, click the Pause button in the top-right of the Logs Explorer. If you have live tail paused and you open a specific log for viewing, a downward facing arrow displays to the far right of the search field.

  • To resume the live tail, click the Play button.

Show details for a logged entity

Click the three vertical dots at the beginning of a log item line and click the IP address or sender's hostname to show only log entries related to the selected IP address or sender's hostname.

Share or bookmark logs

Click the three vertical dots at the beginning of a log item line, and then click the option you want to use:

  • Copy Log Line Text (helpful if you need to paste text elsewhere)
  • Copy Permalink (helpful if you need to bookmark a log or share with users who can log in to SolarWinds Observability)

Show velocity graph

Click the graph button to see patterns in the number of logs received at specific times.

Velocity graphs help identify patterns or anomalies, reducing time spent identifying trends or troubleshooting errors. You can also use them to seek to a particular time by clicking a point on the graph.

By default, the graph shows logs received in the past hour. You can change the time setting in the drop-down list in the top-right of the graph. The number to the left of the time setting drop-down list shows the total number of logs received for the selected period.

  • To see the number of logs received at a specific time in the graph, hover over the graph.

  • To refresh the graph, use the refresh button in the top left of the graph to refresh manually, or select Auto refresh.

Set display preferences

Setting display preferences lets you determine what display best meets your needs. Click the cog button to set your preferences. You can:

  • Change the font or set the density of text displayed in logs.

  • Enable Highlight Matches if you want the text that matches your search to be highlighted.

  • Enable Truncate Message to display each message on one line only. This is great for vertically aligning messages so you can view patterns. Click on a line to expand it.

  • Enable Use UTC Timestamps to convert your log timestamps to UTC (Universal Time Coordinated) without the need to modify the time zone in your Profile.

  • Use the other options to hide information, collapse information to a colored bullet you can hover over to see the full set of information, or show extra information, such as time, system, and severity. Severity represents the severity of events as identified by the application that sent them.

The UTC Timestamps option is not available on your Display Preferences menu if your profile’s time zone is already set to UTC. This option also does not convert the internal timestamps of messages.

Use advanced search syntax

AND

By default, searches containing multiple terms in the same search automatically use AND. For example, this search:

something somethingelse

is equivalent to this search:

something AND somethingelse

OR

To match one or more of multiple terms, use the OR keyword. For example:

something OR somethingelse
OR must be capitalized.

Phrases

Enclose phrases in single- or double-quotes. For example:

something OR "something else"

Single- and double-quotes are interchangeable, as long as the same type of quotation mark is used to start and end the same phrase. To search for a phrase that contains a quotation mark, enclose the phrase in the other type of quotation mark. For example, to match this punctuation-filled phrase:

{"action"=>"index",

enclose it in single-quotes, like this:

'{"action"=>"index",'

Order of operations

To indicate order of operations for matching, use parentheses. For example:

something ("something else" OR "a third thing")

tells SolarWinds Observability to match either of the two terms inside the parentheses, then match (AND) to a term outside the parentheses.

By contrast, the same search without parentheses:

something "something else" OR "a third thing"

is ambiguous about whether the OR applies to just the middle term or to both of the first two terms.

Without parentheses, terms are matched left to right. AND and OR can be used freely inside and outside of parentheses.

Exclusion (negation)

To exclude, prepend a hyphen to the term, quoted phrase, or parenthesized set. For example, to show all logs except a certain word or quoted phrase:

-"exclude this"

Exclude all matches in a parenthesized set:

something -("but not" OR "something else")

Exclusions can be used anywhere in a query. For example:

something -("but not" OR "something else") "but this too"

Case sensitivity

Searches are case insensitive. This query:

Completed in

is equivalent to this query:

completed in

Attributes

SolarWinds Observability supports five attributes: sender, program, message, severity, and facility. All are used in key:value fashion. An example search that constrains results to those from the ssh program containing “something bad” but not “noise”, or those with an error severity, is:

("something bad" program:ssh -noise) OR severity:error

It is possible to include more than one valid value for an attribute. Messages matching either one (OR) are returned:

program:(raid5toolsethtool)

Severity and facility attributes are those from the syslog specification. If severity is a part of the displayed log message, use a text search like error or INFO to find those messages.

Use attributes:

  • To constrain matching to only a single field, either to eliminate false positives or search less data (and increase speed)

  • To search facility or severity, which are not normally examined

Attributes and attribute-less constraints can be used together, and can be combined with negation.

All attributes except message can only have a single value per message, so AND is never relevant. Because of this, all attributes default to OR, as in the program: example above. Program:(a b) means program:(a OR b).

Additionally, any of the attribute names host, source, or system can be used interchangeably with sender.

facility and severity support substring matching, so matches would occur with any facility value beginning with local:

facility:local

As a result, they also support shorthand. There is no need to type the full severity values:

severity:(critemerg)

Punctuation

Other than quotes, you do not need special adaptations to use punctuation in a term or phrase. For example, to see all occurrences of IP addresses within the range 10.20.30.0 to 10.20.30.255, use this query:

10.20.30.

The dots (periods) are interpreted as part of the string. Other punctuation works the same way, and no adaptation or escaping is needed.

Similarly, the entire log message is considered for matching. The search:

bcd

will match lines containing any of these:

abcde
123bcd456
456bcd
456abcde

Sender name and program name

While sender name and program name/filename can be used in searches, they aren’t part of the phrase that SolarWinds Observability matches against. For example, this log message:

www42 httpd: GET / HTTP/1.0

would be matched by any of these queries:

www42 GET
httpd GET
www42 httpd GET
www42 httpd "GET /"
www42 OR "GET /"

However, that message would not be matched by this phrase query because the program name and separating colon are not included in the phrase:

"httpd: GET"

Use JSON search syntax

In addition to using the Google-like search syntax to find things in your logs, Logs Explorer can parse a JSON object that appears at the end of a log line. Each line can contain arbitrary string data before the JSON. For example:

2019-12-02 03:04:05 DEBUG {"a":123,"b":456}

Root level search

json.orgId:1193

Example matches:

Exact match { "orgId": 1193 }
Substring match { "orgId": 11933962 }

Nested search

json.user.name:pete

Example matches:

Exact match { "user": {"name": "Pete"} }
Substring match { "user": {"name": "Peter" } }

Exact Match

json.orgId:"11933962"

Example matches:

Exact match { "orgId": 11933962 }

Negation

json.cursor.tail:false AND -json.orgId:15884562

Example matches:

Different value for orgId { "orgId": 11933962, "cursor": {"tail": false} }
orgId not present { "cursor": {"tail": false} }