Search Query Language

Loggly’s search query language is based on Apache Lucene. Review the following sections below to start putting your data to work:

• Terms
• Operators
• Grouping
• Field Names
• Tokenization
• Wildcards
• Regular Expressions
• Non alpha-numeric symbols
• Case Sensitivity
• Time Ranges
• Source Groups
• Filters
• Advanced
• Tricks & Tips

The Basics

Your First Search

Leave the search box blank and click Search. This way, you’ll see a full picture of what log data exists within Loggly.

Terms

Simple strings, such as foo and 867 phrases, or a local use 6 phrase can be entered directly into the search box. Terms can be combined together with Boolean operators to form a more complex query.

Operators

When using more than one search term in your query, you’ll need to join them together with an operator. Loggly uses the Boolean operators AND, OR, and NOT as well as a final operator TO.

Operator	Description
AND	The intersection of terms. Both terms must exist in the event.
OR	The union of terms. Either term may exist in the event.
NOT	This term must not exist in the event.
TO	Used when searching for ranges of values. A query term using the TO operator must take the form of <field_name>:[low# TO high#]. The operator TO can be used with numeric values as well as words that have lowercase letters, for example, syslog.severity:[informational TO warning]. A field name must be supplied.

The example below would find log events that have both a hostname of frontend01.example.com and a logtype of apache.

syslog.host:frontend01 AND logtype:apache 

The example below would find log events that have either a hostname of frontend01.example.com or frontend02.example.com.

syslog.host:frontend01 OR syslog.host:frontend02 

The example below would find log events that have frontend01.example.com, but not frontend02.example.com.

syslog.host:frontend01 NOT syslog.host:frontend02 

The example below would find response times that are between 100ms and 500ms.

json.responseTime:{100 TO 500} 

The example below would find response times that are greater than 500ms.

json.responseTime:{500 TO *} 

As an alternative to the Boolean operators above, you can also use the plus (+) or minus (-) symbol in place of AND and NOT respectively. These symbols must be used before the query terms they reference without spaces. For example, these search queries are equivalent:

"user list" +micheletto -rufio 

"user list" AND micheletto NOT rufio 

"user list" micheletto NOT rufio 

A space is interpreted as the AND operator by the Loggly engine. The full text search error 404 is interpreted as error AND 404. SolarWinds does not recommend using this shortcut for any but the shortest search queries as it can quickly become hard to decipher search terms separated by only a space in long queries.

Grouping

Query terms can be grouped using parentheses. Terms in parentheses are processed first. This example will find events with a term user list that includes either micheletto or rufio.

"user list" (micheletto OR rufio)

Field Names

The full topic of structured vs unstructured data is covered in more detail in the Log Types section, but here is more information in the context of search.

All logs, both structured and unstructured, have a full-text index. Meaning, every string within your data is searchable.

Additionally, supported log types are parsed, which yields field names and field values from your logs. This means you can search for particular strings or ranges of numerical field values within specific field names. Each field name is pre-pended with the log type: apache, json, syslog, etc. For example, if you have apache logs you could perform search queries such as:

apache.status:[500 TO 599] 

apache.referer:google 

syslog.host:frontend* 

If the data includes nested JSON, the field name should be included with dot notation:

json.sport.baseball:Giants 

Using operators and filters strategically can enhance the specificity of your search results.

You can also do an exact match search on field values using elastic search syntax, example: ._facet. This can be useful if there are special characters or others that are important to getting accurate results.

syslog.host._facet:"frontend01.loggly.com" 

Using Wildcards

The wildcard symbols can be used to bring up results with partial search terms entered.

Operator	Description
*	Will match more than one character.
?	Will match just one character.

Valid use of wildcard expansion:

json.level:*
*
json.level:er*
json.level:erro?

Invalid use of wildcard expansion:

*.level:info
*nfo
json.level:*er

Tokenization

When Loggly indexes your content, it does a full text index, which means that every single string and numeric value is available to search across. Each string and numeric value it finds is considered a token. Loggly also breaks tokens when there is a change in capitalization. For example, if it spots a string like:

frontEnd01.example.com 

It’s actually broken apart into tokens. Loggly inserts a space for any non-alphanumeric character and splits apart any numeric from the alphabetic characters. Any of these searches would match the value.

frontEnd
frontEnd01
End
01
example
com
"frontEnd01.example.com"
"frontEnd01 example com"
front*

Regular Expression Searches

To further power your searches, try using regular expressions to match different patterns. Include your regex between forward slashes( /). For example, to search for either wall or ball, the following would work:

/[wb]all/

To search for more than four consecutive zeros:

/0{4,}/

Regular expressions only work on single tokens when run on a full text search. You cannot do a regex on a single term that has been split into several tokens due to special characters or other rules. For example, the example below wouldn't work because it has been tokenized, even escaping escape the special characters:

/HTTP/1.0/

Instead, consider doing a search on ordered tokens by quoting the search term.

"HTTP/1.0"

You can also do an exact match search using an elastic search ._facet notation. This is useful if you want to search without Tokenization or if special characters are important in your search.

syslog.host._facet:"frontend01.loggly.com" 

For further details on the regular expression syntax Loggly supports, review Lucene’s Regular Expression class.

Non alpha-numeric symbols in Search

Only alpha-numeric characters will be matched within log events. Special characters + – & || ! ( ) { } [ ] ^ " ~ * ? : _ cannot be matched and should not be included as part of the search query. If you do include special characters within your search query, be sure to use quotes around each term. For example, an unquoted colon will cause an error if it’s not preceded by a valid field name.
These two searches will show equivalent results:

"https://yourwebsite.com:9000" 

"http yourwebsite com 9000" 

This search will result in an invalid query since the colon is not quoted:

https://yourwebsite.com:9000 

Case sensitivity

Loggly is case-insensitive to all terms in a search query EXCEPT in two cases:

• The TO operator
• Field names, including tag and logtype

Time Ranges in Search

Time ranges can be relative or absolute. Relative time ranges are pegged to the current time. Quick picks for some relative ranges are available. To supply an absolute time range or a custom relative range, choose Custom time range from the date selector to see the option:

Custom Time Range Syntax

Type	Description	Example
Absolute Dates	yyyy-MM-dd HH:mm:ss.SSS	2013-08-11 00:00:00.000
	yyyy-MM-ddTHH:mm:ss.SSSZ	2013-08-13T18:45:00.000Z
Relative Dates	now	now
	Last 5 weeks	-5w
	Last 12 hours	-12h
	Last 5 minutes	-5m
	Last 30 seconds	-30s

Source Groups

One source group can be provided per search. By narrowing down the search criteria before you perform the search, you can be assured that your search will return relevant results. For example, you may want to search just your staging logs for an event that happened on your application servers. If you include a set of hosts called staging* and including a tag called python, you can recall this Source Group when necessary. Source Groups are optional groupings, but very useful for segmenting large amounts of disparate data. Source Groups are fully explained later in this document.

Filters

Filters are another method of narrowing down your search query. The Filter by Field section on the left side of your search screen can show a list of fields. Each of these field names, when clicked, will show the list of top values beneath. Click Show All to see all of the top values for each field. The Filter by Field section is explained later in this document.

Advanced Queries

Loggly also supports several types of Lucene- or Elastic- search based queries.

Proximity match allows you to search for tokens within a certain distance of each other. For example, the example below will find the tokens user and root within a distance of 4 tokens.

"user root"~4

Fuzzy match allows you to search for a token within a character edit distance of 2. For example, this will match fuzzy or fuzzy1.

fuzz~

Exists query allows you to find all events that contain a specific field. For example, this will match all events that contain json.level field:

_exists_:json.level

When you want to find all events that do not contain the field you just add a NOT operator in front of the query:

NOT _exists_:json.level

Search Tips and Tricks

Here are some things to keep in mind while searching if you’re not seeing the results you expect:

1. Clear Source Groups.

Change your Source Group to All Sources.

2. Quote or avoid special characters.

Special characters: + – & || ! ( ) { } [ ] ^ " ~ * ? : / \ 

Many of these characters are used by Loggly’s query language, so they shouldn’t be part of your query term.

3. Expand your time range.

Check your time range,. You may not have noticed that you’re searching for seconds instead of hours.

4. Remove filters and terms.

Check for the pills under the search box. Reduce the number of terms you’ve included.

5. Check your spelling

Watch out for typos---both in your log event and in the query.

6. Don’t use partial strings without a wildcard.

Here’s an example log message:

Timer-2 S3Stats.processStatEvents INFO: Processed stats for Stats-6297-138

That event will be indexed by the following strings:

timer 2 S 3 stats process stat events info processed stats for stats 6297 138

Because of that, the following terms will match the event:

timer
process
event*

These terms will not match the event:

event
time