Discover

You can explore your data interactively from the Discover page. You have access to every document in every index which matches the selected index pattern. You can submit search queries, filter the search results, and view document data. You can also see the number of documents that match the search query and get field value statistics. If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in a histogram at the top of the page.

Discover.

Setting the time filter

The time filter restricts the search results to a specific time period. You can set a time filter if your index contains time-based events and a time-field is configured for the selected index pattern.

Because of the variety of date formats in use, and the possibility that an index is fixed to use a specific format, Siren provides sample valid dates when editing filters.

By default, the time filter is set to the last 15 minutes. You can use the Time Picker to change the time filter or select a specific time interval or time range in the histogram at the top of the page.

To set a time filter with the Time Picker:

  1. Click Time Picker (image) in the Siren Investigate toolbar.

  2. To set a quick filter, click one of the shortcut links.

    Time filter shortcuts.

  3. Click Select All in the Apply to Dashboards section to apply the time filter to all dashboards. Alternatively, you can select individual dashboards. The time filter is applied to the current dashboard by default.

  4. To specify a time filter relative to the current time, click Relative and specify the start time as a number of seconds, minutes, hours, days, months or years. You can also specify the end time relative to the current time. Relative times can be in the past or the future.

    Relative time filter.

  5. To specify both the start and end times for the time filter, click Absolute and select a start and end date. You can adjust the time by editing the To and From fields.

    Absolute time filter.

  6. Click the caret (image) in the bottom right corner to close the Time Picker.

To set a time filter from the histogram, do one of the following:

  • Click the bar that represents the time interval you want to zoom in on.

  • Click and drag to view a specific time span. You must start the selection with the cursor over the background of the chart. The cursor changes to a plus sign (+) when you move the mouse pointer over a valid start point.

To move forward or backward in time, click the arrows to the left or right of the Time Picker:

Move backwards in time.

You can use your browser’s Back button to undo your changes.

The displayed time range and interval are shown on the histogram. By default, the interval is set automatically based on the time range. To use a different interval, click the link and select an interval.

Searching your data

You can search the indices that match the current index pattern by entering your search criteria in the Query bar. You can perform a simple text search, use the Lucene query syntax, or use the full JSON-based Elasticsearch Query DSL.

When you submit a search request, the histogram, Documents table, and Fields list are updated to reflect the search results. The total number of hits (matching documents) is shown in the toolbar. The Documents table shows the first five hundred hits. By default, the hits are listed in reverse chronological order, with the newest documents shown first. You can reverse the sort order by clicking the Time column header. You can also sort the table by the values in any indexed field. For more information, see Viewing document data.

To search your data, enter your search criteria in the Query bar and press Enter or click Search (image) to submit the request to Elasticsearch.

  • To perform a free text search, enter a text string. For example, if you are searching web server logs, you could enter safari to search all fields for the term safari.

  • To search for a value in a specific field, prefix the value with the name of the field. For example, you could enter status:200 to find all of the entries that contain the value 200 in the status field.

  • To search for a range of values, you can use the bracketed range syntax, [START_VALUE TO END_VALUE]. For example, to find entries that have 4xx status codes, you could enter status:[400 TO 499].

  • To specify more complex search criteria, you can use the Boolean operators AND, OR, and NOT. For example, to find entries that have 4xx status codes and have an extension of php or html, you could enter status:[400 TO 499] AND (extension:php OR extension:html).

These examples use the Lucene query syntax. You can also submit queries using the Elasticsearch Query DSL. For examples, see query string syntax in the Elasticsearch Reference.

Saving searches enables you to reload them into Discover and use them as the basis for visualizations. Saving a search saves both the search query string and the currently selected index pattern.

To save the current search:

  1. Click Save in the Siren Investigate toolbar.

  2. Enter a name for the search and click Save.

You can import, export and remove saved searches from Management > Saved Objects.

To load a saved search into Discover:

  1. Click Open in the Siren Investigate toolbar.

  2. Select the search you want to open.

If the saved search is associated with a different index pattern than is currently selected, opening the saved search also changes the selected index pattern.

Changing which indices you are searching

When you submit a search request, the indices that match the currently-selected index pattern are searched. The current index pattern is shown below the toolbar. To change which indices you are searching, click the index pattern and select a different index pattern.

For more information about index patterns, see Creating an Index Pattern Search.

Refreshing the search results

As more documents are added to the indices you are searching, the search results shown in Discover and used to display visualizations get stale. You can configure a refresh interval to periodically resubmit your searches to retrieve the latest results.

To enable auto refresh:

  1. Click Time Picker (image) in the Siren Investigate toolbar.

  2. Click Auto refresh.

  3. Choose a refresh interval from the list.

    Auto refresh intervals.

When auto refresh is enabled, the refresh interval is displayed next to the Time Picker, together with a Pause button. To temporarily switch off auto refresh, click Pause.

If auto refresh is not enabled, you can manually refresh visualizations by clicking Refresh.

Filtering by field

You can filter the search results to display only those documents that contain a particular value in a field. You can also create negative filters that exclude documents that contain the specified field value.

You can apply field filters in the following ways:

  • From the Selected fields list

  • From the Documents table

  • By manually adding a filter

The applied filters are shown below the Query bar. Negative filters are shown in red.

To add a filter from the Selected fields list:

  1. Click the name of the field you want to filter on. This displays the top five values for that field.

    filter field

  2. Click one of the following filters:

    • The Filter for value (image) button includes only those documents that contain that value in the field.

    • The Filter out value (image) button excludes documents that contain that value in the field.

To add a filter from the Documents table:

  1. Click the Expand icon to the left of the document’s table entry.

    Expand icon

  2. In the expanded view, click one of the following filters:

    Expanded Document

    • Filter for value includes only those documents that contain that value in the field.

    • Filter out value excludes documents that contain that value in the field.

    • Toggle column in table allows you to hide or show a field as a column in the table.

    • Filter for field present includes only those documents that contain the field.

    • Filter for more like this includes documents that contain similar text to the text in that field.

To manually add a filter:

  1. Click Add a filter. A popup will be displayed for you to create the filter.

    Add filter.

  2. Choose a field to filter by. This list of fields will include fields from the index pattern you are currently querying against.

    Add filter field.

  3. Choose an operation for your filter.

    Add filter operator.

    The following operators can be selected:

    is Filter where the value for the field matches the given value.

    is not

    Filter where the value for the field does not match the given value.

    is one of

    Filter where the value for the field matches one of the specified values.

    is not one of

    Filter where the value for the field does not match any of the specified values.

    is between

    Filter where the value for the field is in the given range.

    is not between

    Filter where the value for the field is not in the given range.

    exists

    Filter where any value is present for the field.

    does not exist

    Filter where no value is present for the field.

  4. Choose the value(s) for your filter.

    add filter value

  5. (Optional) Specify a label for the filter. If you specify a label, it will be displayed below the Query bar instead of the filter definition.

  6. Click Save. The filter will be applied to your search and be displayed below the Query bar.

To make the filter editor more user-friendly, you can enable the filterEditor:suggestValues advanced setting. Enabling this will cause the editor to suggest values from your indices if you are filtering against an aggregatable field. However, this is not recommended for extremely large data sets, as it can result in long queries.

Managing filters

To modify a filter, move the moue pointer over it and click one of the action buttons.

Filter action buttons.

image Enable Filter

Switch off the filter without removing it. Click again to switch the filter on again. Diagonal stripes indicate that a filter is switched off.

image Pin Filter

Pin the filter. Pinned filters persist when you switch contexts in Siren Investigate. For example, you can pin a filter in Discover and it remains in place when you switch to Visualize. Note that a filter is based on a particular index field—if the indices being searched do not contain the field in a pinned filter, it has no effect.

image Invert Filter

Switch from a positive filter to a negative filter and vice-versa.

image Remove Filter

Remove the filter.

image Edit Filter

Edit the filter definition. Enables you to manually update the filter and specify a label for the filter.

To apply a filter action to all the applied filters, click Actions and select the action.

Editing a filter

You can edit a filter by changing the field, operator, or value associated with the filter (see the Add Filter section), or by directly modifying the filter query that is performed to filter your search results. This enables you to create more complex filters that are based on multiple fields.

  1. To edit the filter query, first click Edit for the filter, then click Edit Query DSL.

    Edit filter query.

  2. You can then edit the query for the filter.

    Edit filter query JSON.

For example, you could use a bool query to create a filter for the sample log data that displays the hits that originated from Canada or China that resulted in a 404 error:

{
  "bool": {
    "should": [
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "China"
        }
      }
    ],
    "must": [
      {
        "term": {
          "response": "404"
        }
      }
    ]
  }
}

Viewing document data

When you submit a search query, the 500 most recent documents that match the query are listed in the Documents table. You can configure the number of documents shown in the table by setting the discover:sampleSize property in Advanced Settings. By default, the table shows the localized version of the time field configured for the selected index pattern and the document _source. You can add fields to the Documents table from the Fields list. You can sort the listed documents by any indexed field that’s included in the table.

To view a document’s field data, click the Expand icon to the left of the document’s table entry.

image

To view the original JSON document (pretty-printed), click the JSON tab.

To display or hide a field’s column in the Documents table, click Toggle column in table (image).

To collapse the document details, click Collapse (image).

Sorting the document list

You can sort the documents in the Documents table by the values in any indexed field. If a time field is configured for the current index pattern, the documents are sorted in reverse chronological order by default.

To change the sort order, move the mouse pointer over the name of the field you want to sort by and click the sort button. Click again to reverse the sort order.

Adding field columns to the documents table

By default, the Documents table shows the localized version of the time field that’s configured for the selected index pattern and the document _source. You can add fields to the table from the Fields list or from a document’s field data.

To add a field column from the Available fields list, move the mouse pointer over the field and click Add.

Add Field From Sidebar

To add a field column from a document’s field data, expand the document and click the field’s Toggle column in table (image) button.

Added field columns replace the _source column in the Documents table. The added fields are also added to the Selected fields list.

To rearrange the field columns, move the mouse pointer over the header of the column you want to move and click the Move left or Move right button.

Move column.

Removing field columns from the documents table

To remove a field column from the Documents table, move the mouse pointer over the header of the column you want to remove and click Remove (image).

Viewing document context

For certain applications it can be useful to inspect a window of documents surrounding a specific event. The context view enables you to do just that for index patterns that are configured to contain time-based events.

To show the context surrounding an anchor document, click the Expand icon to the left of the document’s table entry and then click the View surrounding documents link.

image

Expanded document

The context view displays a number of documents before and after the anchor document. The anchor document itself is highlighted in blue. The view is sorted by the time field specified in the index pattern configuration and uses the same set of columns as the Discover view the context was opened from. If there are multiple documents with the same time field value, the internal document order is used as a secondary sorting criterion by default.

The field used for tie breaking in case of equal time field values can be configured using the advanced setting context:tieBreakerFields in Management > Advanced Settings, which defaults to the _doc field. The value of this setting can be a comma-separated list of field names, which will be checked in sequence for suitability when a context is about to be displayed. The first suitable field is then used as the tie breaking field. A field is suitable if the field exists and is sortable in the index pattern the context is based on.

While not required, you should only use fields which have doc values enabled to achieve good performance and avoid unnecessary field data usage. Common examples for suitable fields include log line numbers, monotonically increasing counters and high-precision timestamps.

Context view.

The number of documents displayed by default can be configured using the context:defaultSize setting in Management > Advanced Settings.

Changing the context size

You can change the number documents displayed before and after the anchor document independently.

To increase the number of displayed documents that are newer than the anchor document, click Load 5 more above the document list or enter the desired number into the input box.

image

To increase the number of displayed documents that are older than the anchor document, click Load 5 more below the document list or enter the desired number into the input box

image

The default number of documents loaded with each click can be configured using the context:step setting in ManagementAdvanced Settings.

Filtering the context

Depending on how the documents are partitioned into index patterns, the context view may contain many documents not related to the event under investigation. To adapt the focus of the context view to the task at hand, you can use filters to restrict the documents considered by Siren Investigate for display in the context view.

When switching from the Discover view to the Context view, the previously applied filters are carried over. Pinned filters remain active while normal filters are copied in a switched off state. You can selectively re-enabled them to refine your context view.

New filters can be added using the Add a filter link in the filter bar, by clicking the filter icons appearing when moving the mouse pointer over a field, or by expanding documents and clicking the filter icons in the table.

Discover context view filter montage.

Viewing field data statistics

From the Fields list, you can see how many of the documents in the Documents table contain a particular field, what the top five values are, and what percentage of documents contain each value.

To view field data statistics, click the name of a field in the Fields list.

Field statistics.

Selected fields

Selected Fields are displayed on the Selected Field list at the top left of the Discover page.

Selected fields.

Click a field to see the field’s Viewing field data statistics.

To remove a field, click the field and click Remove.

Remove selected field.

After you have selected at least one field, you can then click Generate Dashboard to begin auto generating a new Dashboard.

If you have not selected any fields, you can allow Siren Investigate to select the fields it believes are the most relevant by clicking Autoselect Most Relevant.

This button is only visible if there are no fields selected. If you have selected fields, the Generate Dashboard button will be visible in its place.

Auto select fields

Siren Investigate begins testing the fields with each field being analyzed for relevance against a number of heuristics, for example, whether all values are unique - which indicates a potential ID field and unlikely to be relevant for visualizations.

Auto select report.

After all the fields have been analyzed, a test report is displayed.

Auto select report.

This report shows all the fields in the Discover data, which fields are selected as most relevant, the field type, the visualization selected for that type, the relevancy score and any notes on why the field was or was not selected as a relevant field.

You can add and remove fields you would like selected using the check boxes on the left.

When you are ready, click Ok to select the fields.

After the fields have populated the Selected Fields list, you are ready to generate a dashboard.