If you've used Google search, you're familiar with 95% of Papertrail's search syntax. Type what you have, whether it's an IP address, an error string, the name of a program or system, a user's session, or fragments of any of those things.
From there, build up sets of what you do and don't want to see. Here's a few examples for the Ruby on Rails framework to give you an idea.
This came from our experience using Papertrail for our logs, from watching early users try different syntaxes, and most importantly, from seeing what we and others actually wanted to find. By far the most common type of search was a set of desired program names or message strings (important things) combined with a few exclusions (noise).
This Boolean syntax makes queries like that fast to compose and easy to read/update.
Check out Linking to logs to quickly integrate Papertrail searches into your app’s internal admin interface.
These 6 searches demonstrate most of the syntax. The sender name, program/file name, and message are all searchable:
"Accepted password" sudo
(www42 some.log) OR (nyc-db2 other.log)
(message-id postfix/smtpd) OR -google.com
10.1.2 -("TTL exceeded" OR "1 packet")
Searches can also be constrained by an attribute, so Papertrail considers only that field:
program:(production.log staging.log) '"action"=>"index"' -("count(*)" OR abc)
For particulars on searching JSON objects please refer to the JSON Search Syntax article.
While in the event viewer, click the question mark icon on the right side of the search box to see these examples without leaving the viewer:
To get the narrowest set of results as fast as possible:
Create groups that represent different sets of systems, then search those groups (instead of searching an all-encompassing group). Aside from making search faster, properly-scoped groups make the Dashboard easier to interpret, since searches are tied to the group they’re actually related to rather than being lumped together. As an example,
Production Web serversmay be far fewer logs than
Production servers, and when a search is specific to Web logs, it probably logically belongs there.
Use attributes when searching only specific parts of a log message. For example,
abc sender:www42will be faster than
Provide (or link with) a time/date or rough estimate. Papertrail will start searching from the time you seek to in the event viewer. Seek to a time at the end of or after the incident so that the search starts there.
Integrate contextual "Show related logs" links from your app's admin dashboard. Your dashboard can generate links to logs with timestamps (such as a timestamp from a user’s last login or when a background job completed) and search queries. The additional search keywords (beyond what you would manually type) may return fewer unrelated matches and it saves time entering a search.
Papertrail supports combining terms and phrases using AND, OR, and exclusion operators, as well as order of operations (parentheses). The message text, program/file name, and system name (as shown in Papertrail's dashboard) are searched.
By default, terms in the same search are automatically ANDed. This search:
is equivalent to this search:
something AND somethingelse
To match one or more of multiple terms, use the
OR keyword. For example:
something OR somethingelse
OR must be capitalized.
Enclose phrases in single- or double-quotes. For example:
something OR "something else"
Single- and double-quotes are interchangeable, as long as the same type of quote is used to start and end the same phrase. To search for a phrase that contains a quote, use the other type of quote. For example, to match this punctuation-filled phrase:
enclose it in single-quotes, like this:
Backslashes can be used to escape single- and double-quotes within words or phrases. Therefore, to find a literal backslash, it must be doubled up, as in
To indicate order of operations for matching, use parentheses. For example:
something ("something else" OR "a third thing")
says to match either of the 2 terms inside the parentheses, then
AND it with the term outside the parentheses.
Contrast that search with the same search without parentheses:
something "something else" OR "a third thing"
which is ambiguous about whether the
OR applies to just the middle term, or to both of the first 2 terms. (Without parentheses, terms are matched left to right).
OR can be freely used inside and outside of parentheses.
- to a term, quoted phrase, or parenthesized set. For example,
to show all logs except a certain word or quoted phrase:
Exclude all matches in a parenthesized set:
something -("but not" OR "something else")
Exclusions can be used anywhere in a query. For example:
something -("but not" OR "something else") "but this too"
Searches are case insensitive. This query:
is equivalent to this query:
Papertrail supports 5 attributes: sender, program, message, severity, and facility. All are used in
key:value fashion. An example search that constrains results to those from the
ssh program containing "something bad" but not "noise", or those with an error severity, is:
("something bad" program:ssh -noise) OR severity:error
It's possible to include more than one valid value for an attribute. Messages matching either one are returned (OR):
Severity and facility attributes are those from the syslog specification. If severity is a part of the displayed log message, simply use a text search like
INFO to find those messages.
- to constrain matching to only a single field, either to eliminate false positives or search less data (and increase speed)
- to search facility or severity, which are not normally examined
Attributes and attribute-less constraints can be used together, and can be combined with negation.
All attributes except
message can only have a single value per message, so AND is never relevant. Because of this, all attributes default to OR, as in the
program: example above.
program:(a b) means
program:(a OR b).
Additionally, any of the attribute names
system can be used interchangeably with
severity support substring matching, so this matches any
facility value beginning with
As a result, they also support shorthand, like this. No need to type the full
For more examples, see Papertrail's attribute announcement.
Other than quotes and backslashes, no special adaptations should be needed to use punctuation in a term or phrase. For example, to see all occurrences of IP addresses within
10.20.30.255, use this query:
The periods are interpreted as part of the string. Other punctuation works the same way, and no adaptation or escaping is needed.
Similarly, the entire log message is considered for matching. The search:
will match lines containing any of these:
To search for a time or date, click the clock icon to the right of the search box and enter a time, date, or combination. The time or date can be in almost any format, and will be parsed and displayed in realtime. Here's a screenshot:
When searching by time in the Web interface, the searched-for timestamp is interpreted in the timezone set in the user profile.
Papertrail tracks the time a log message was received, and automatically displays it in the profile timezone. Users can have different timezones within the same organization, so there’s less need to perform mental timezone conversions.
While sender name and program name/filename can be used in searches, they aren't part of the phrase that Papertrail matches against. For example, this log message:
www42 httpd: GET / HTTP/1.0
would be matched by any of these queries:
www42 httpd GET
www42 httpd "GET /"
www42 OR "GET /"
However, that message would not be matched by this phrase query because the program name and separating colon are not included in the phrase:
This document has moved to Linking to logs.
The scripts are not supported under any SolarWinds support program or service. The scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.