Introduction

Kibi is an open source data intelligence platform built upon Kibana 4.6.4.
Current Kibi v4.6.4-3 support elasticsearch v2.4.4 and Siren Join Plugin version v2.4.4.

Kibi allows to perform complex analytics on large volumes of data by providing customizable visualizations (charts, maps, metrics and tables) on Elasticsearch queries; visualizations can be organized into multiple dashboards, presented in a tabbed user interface.

Elasticsearch results can be filtered and augmented by queries on multiple external datasources such as SQL databases and triplestores; queries on external datasources can also be used as aggregations in visualizations.

In addition to standard Kibana visualizations, Kibi provides:

  • The Relational Filter visualization, which allows to configure relations between fields in different indices and apply cross-dashboard filters (pivoting).

  • The Relational Panel, which defines relations between indices by specifying on which fields to join. This enables to browse dashboards as a connected set, where filtering one dashboards updates results from the other dashboards.

  • The Kibi Word Cloud visualization, which displays a cloud of high frequency terms from the documents returned by an Elasticsearch query.

  • The Kibi Timeline visualization, which displays a timeline with multiple groups of data coming from different indices.

  • The Radar Chart visualization, which is a graphical method for displaying multivarate data. with multiple groups of data coming from different indices.

  • The Bubble Diagram visualization, which displays series of data grouped into packed circles.

  • The Scatter plot visualization, which displays a scatter plot chart in different modes.

  • The Box plot visualization, which displays a box plot chart from the data in the current set of Elasticsearch documents.

  • The Kibi Vector Map visualization, which displays a vector map where countries are colored based on aggregated field value.

  • The Kibi Horizontal Bar Chart visualization, which displays a horizontal bar chart.

  • The Multi chart visualization, which displays a different type of charts for the same data and allow save and select multiple aggregation configurations.

  • The Enhanced search results visualization, which displays query results in a table.

  • The Kibi Query Viewer, which enables the visualization of queries on external datasource through Jade or Handlebars templates.

  • The Kibi Graph Browser, which displays the currently selected Elasticsearch documents as a node of a graph and allows the user to visually explore the connection between vertices.

The Relational Filter visualization requires the Siren Join plugin 2.4.4 for Elasticsearch. The plugin is compatible with Elasticsearch 2.4.4.

How does Kibi compare to Kibana?

Kibi is currently developed as a fork of Kibana 4.6.4. The plugin API of Kibana 4.2.x allows us to keep the amount of forked code to a minimum, but as the API is currently in development, a fork is still needed.

What’s new in Kibi v4.6.4-3

This new version of Kibi provides many improvements and new features. To see all changes check full release notes

Kibi license [Enterprise Edition only]

NOTE: Documentation of Kibi license plugin is available only in Kibi Enterprise Edition.

Setup

Download and install Java 1.8 or greater from https://java.com/en/download/; if you’re on a GNU/Linux distribution, install Java through your package manager.

After Java has been installed, make sure that the java command is available in the system path by opening a terminal window and running java -version; the command should display the version of Java installed on your system.

You can set up Kibi and start exploring your Elasticsearch indices in minutes. All you need is:

  • Elasticsearch 2.4.x - Elasticsearch compability table

  • A modern web browser - Supported Browsers.

  • Information about your Elasticsearch installation:

    • URL of the Elasticsearch instance you want to connect to.

    • Which Elasticsearch indices you want to search.

Note
If your Elasticsearch installation is protected by Shield see Shield with Kibana 4 for additional setup instructions.

Download the Kibi binary distribution from http://siren.solutions/kibi and extract it to a local directory.

To start the Elasticsearch cluster included in the distribution, open a terminal window, change to the directory where you extracted Kibi and run the following command:

$ cd elasticsearch/
$ ./bin/elasticsearch

If you’re on Windows, run the following command:

$ cd elasticsearch\
$ .\bin\elasticsearch.bat

To start Kibi, open a second terminal window, change to the directory where you extracted Kibi and run the following command:

Kibi and Elasticsearch Dynamic Mapping

By default, Elasticsearch enables dynamic mapping for fields. Kibi needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .kibana index where saved searches, visualizations, and dashboards are stored.

If your Elasticsearch use case requires you to disable dynamic mapping, you need to manually provide mappings for fields that Kibi uses to create visualizations. You also need to manually enable dynamic mapping for the .kibana index.

The following procedure assumes that the .kibi index does not already exist in Elasticsearch and that the index.mapper.dynamic setting in elasticsearch.yml is set to false:

  1. Start Elasticsearch.

  2. Create the .kibana index with dynamic mapping enabled just for that index:

    $ cd kibi/
    $ ./bin/kibi

If you’re on Windows, run the following command:

$ cd kibi\
$ .\bin\kibi.bat

Upon first launch, the Windows firewall might display the following dialog box; to allow the Kibi backend to connect to external datasources, please check both check boxes (as shown below) and press the Allow access button.

Windows firewall warning

Kibi can be accessed by visiting http://localhost:5606 or for installation with Shield visiting https://localhost:5606; an overview of the included demo dataset is available in Getting started section.

If you want to connect Kibi to a different Elasticsearch cluster, open the configuration file in kibi/config/kibi.yml, set the correct URL in the elasticsearch.url parameter, then restart Kibi. Example of a kibi.yml file.

Connect Kibi with Elasticsearch

Before you can start using Kibi, you need to tell it which Elasticsearch indices you want to explore. The first time you access Kibi, you are prompted to define an index pattern that matches the name of one or more of your indices. That’s it. That’s all you need to configure to start using Kibi. You can add index patterns at any time from the Settings tab.

Tip
By default, Kibi connects to the Elasticsearch instance running on localhost. To connect to a different Elasticsearch instance, modify the Elasticsearch URL in the kibi.yml configuration file and restart Kibi. For information about using Kibi with your production nodes, see [production].

To configure the Elasticsearch indices you want to access with Kibi:

  1. Point your browser at port 5606 to access the Kibi UI. For example, http://localhost:5606 or http://YOURDOMAIN.com:5606.

    Kibi start page

  2. Specify an index pattern that matches the name of one or more of your Elasticsearch indices. By default, Kibi guesses that you’re working with data being fed into Elasticsearch by Logstash. If that’s the case, you can use the default logstash- as your index pattern. The asterisk () matches zero or more characters in an index’s name. If your Elasticsearch indices follow some other naming convention, enter an appropriate pattern. The "pattern" can also simply be the name of a single index.

  3. Select the index field that contains the timestamp that you want to use to perform time-based comparisons. Kibi reads the index mapping to list all of the fields that contain a timestamp. If your index doesn’t have time-based data, disable the Index contains time-based events option.

    Warning
    Using event times to create index names is deprecated in this release of Kibi. Support for this functionality will be removed entirely in the next major Kibi release. Elasticsearch 2.1 includes sophisticated date parsing APIs that Kibi uses to determine date information, removing the need to specify dates in the index pattern name.
  4. Click Create to add the index pattern. This first pattern is automatically configured as the default. When you have more than one index pattern, you can designate which one to use as the default from Settings > Indices.

Voila! Kibana is now connected to your Elasticsearch data. Kibana displays a read-only list of fields configured for the matching index.

Note
Kibana relies on dynamic mapping to use fields in visualizations and manage the .kibana index. If you have disabled dynamic mapping, you need to manually provide mappings for the fields that Kibana uses to create visualizations. For more information, see Kibana and Elasticsearch Dynamic Mapping.

Kibi and Elasticsearch Dynamic Mapping

By default, Elasticsearch enables dynamic mapping for fields. Kibi needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .kibi index where all its configuration is stored.

If your Elasticsearch use case requires you to disable dynamic mapping, you need to manually provide mappings for fields that Kibi uses to create visualizations.

You also need to manually enable dynamic mapping for the .kibi index.

The following procedure assumes that the .kibi index does not already exist in Elasticsearch and that the index.mapper.dynamic setting in elasticsearch.yml is set to false:

  • Start Elasticsearch.

  • Create the .kibi index with dynamic mapping enabled:

curl -X PUT <your Elasticsearch URL>/.kibi -d '{ "index.mapper.dynamic": true }'
  • Start Kibi, open the UI in your browser and verify that there are no error messages related to dynamic mapping. For a brief tutorial that explores these core Kibi concepts, take a look at the Getting Started page.

Upgrading from a previous version

An existing Kibi installation can be upgraded as follows:

  • backup the .kibi index.

  • backup the .kibiaccess index if Kibi ACL is enabled.

  • backup the Kibi configuration file (config/kibi.yml)

  • upgrade Elasticsearch; before restarting each node, make sure to install a compatible version of the SIREn Join plugin.

  • download and extract the new Kibi version.

  • copy the previous configuration file to the config directory of the new Kibi version.

  • check for breaking changes to the configuration described below.

  • install the compatible versions of third party Kibi/Kibana plugins that you might need in addition to the bundled ones.

  • execute the upgrade command.

Breaking changes

4.5.4

Enterprise Edition

If you are using Elastic Shield to secure the cluster that Kibi connects to, ensure that the kibi_core.elasticsearch.auth_plugin parameter is set to shield, e.g.:

kibi_core:
  elasticsearch:
    auth_plugin: "shield"

Backing up the .kibi index.

Before upgrading it is strongly recommended to have a backup of the .kibi index; the recommended way to perform regular backups of Elasticsearch indexes is through the snapshot and restore modules.

If Kibi ACL is enabled, include the .kibiaccess index in the backup.

A useful tool to take a one-off backup is elasticdump; once installed, it is possible to dump the .kibi index mappings and data on two separate files by running the following commands:

elasticdump --input=http://<cluster address>:<cluster port>/<kibi index> --type=mapping --output=<mappings_dump_file>

elasticdump --input=http://<cluster address>:<cluster port>/<kibi index> --type=data --output=<data_dump_file>

Full example:

$ elasticdump --input=http://192.168.99.100:9200/.kibi --type=mapping --output=mappings.json
$ elasticdump --input=http://192.168.99.100:9200/.kibi --type=data --output=data.json

If needed, you can restore the index from dumps by running the following commands:

elasticdump --input=<mappings_dump_file> --type=mapping --output=http://<cluster address>:<cluster port>/<kibi index>

elasticdump --input=<data_file> --type=data --output=http://<cluster address>:<cluster port>/<kibi index>

Full example:

$ elasticdump --input=mappings.json --type=mapping --output=http://192.168.99.100:9200/.kibi
$ elasticdump --input=data.json --type=data --output=http://192.168.99.100:9200/.kibi

Upgrading the .kibi index.

To upgrade the objects in the .kibi index (dashboards, visualizations, etc.), move to the directory in which Kibi is installed and execute the following command:

bin/kibi upgrade

The command will look for out of date objects and upgrade them, e.g.:

$ bin/kibi upgrade
  log   [17:58:33.494] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [17:58:36.127] [info][migrations] Executing migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.141] [info][migrations] Executed migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.142] [info][migrations] Executing migration "Upgrade kibi graph browser visualization to version 2."
  log   [17:58:36.157] [info][migrations] Executed migration "Upgrade kibi graph browser visualization to version 2."
  log   [17:58:36.158] [info][migrations] Executing migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executed migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executing migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executed migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executing migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
  log   [17:58:36.400] [info][migrations] Executed migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
Upgraded 20 objects.

It is possible to run the command multiple times, however running the command at the same time from multiple machines is not supported.

Getting Started

The demo distribution

The quickest way to get started with Kibi is to download the Kibi demo distribution from http://siren.solutions/kibi ; the page contains also introductory screencast which we recommend to watch.

The Kibi demo distribution includes a ready to use Elasticsearch cluster in the elasticsearch directory; the cluster contains five indices:

company

a collection of companies

article

a collection of articles about companies

investment

a collection of investments in companies

investor

a collection of investors

.kibi

Kibi configuration

The indices have been populated through Logstash from the SQLite database in kibi/crunchbase.db; the Logstash configuration used to populate the indices is described in the Loading data into Elasticsearch chapter.

Note
The demo dataset has been built from a sample of articles gathered from tech blogs in 2013 and from data about companies, investments and investors in the CrunchBase 2013 Snapshot, which is copyright © 2013 AOL Inc.

The current CrunchBase dataset is available at http://data.crunchbase.com/.

After starting Elasticsearch and Kibi, as described in the Setup chapter, start your web browser and navigate to http://localhost:5606 or for installation with Shield https://localhost:5606.

By default, Kibi displays the Articles dashboard; dashboards can be configured to display multiple visualizations on the documents stored in a specific index or returned by a saved search on an index.

Each dashboard is represented by a tab containing the dashboard title and the number of documents available to visualizations.

The articles dashboard

You can quickly search specific articles through the search input in the navigation bar; for example, let’s find all the articles written about wireless or wifi:

The dashboard search bar

We can immediately see that there are 11693 articles about those topics and all the visualizations are refreshed to aggregate data for this subset of articles.

Note
Besides simple text, queries in the search bar can be written using the Lucene query syntax, or Elasticsearch Query DSL.

Video tutorials are also available:

Filters

Visualizations can be used to create filters; for example, you can see all the articles about wireless or wifi published by TechCrunch by clicking on the TechCrunch slice inside the Articles by Source pie chart visualization:

Clicking on a pie slice
The dashboard with the filter on the slice applied
Note
To disable a filter, just move the mouse over it and click on the checkbox icon; you can disable all the filters applied to the dashboard by clicking on Actions and then Disable; for more information about filtering, please read the filters chapter.

Relational filters

The Relational Filter visualization allows to create cross-dashboard filters; for example, by looking at the Companies button in the dashboard, you can see that there are 342 companies mentioned in the TechCrunch articles about wireless or wifi.

A relational filter button

By clicking on the button, you can switch to the Companies dashboard and visualize the data about these 342 companies:

Relational filter from Articles to Companies

The relational filter created by clicking on the button is displayed in the filter bar, and can be disabled or deleted just like any other filter; moving the mouse over the filter will display the list of joined indices and their filters:

Relational filter in the filter bar

Relational filter can be accumulated; for example, if you click on the Investment rounds -→ button, you will see data about the 298 investment rounds related to a subset of 342 companies mentioned in the TechCrunch articles about wireless or wifi.

Click on the Companies tab to go back to the Companies dashboard.

To understand how to define a relational filter, click the pencil icon inside the Relational widget visualization heading; this will open the configuration editor:

Relational filter configuration

As you can see, it is possible to set two different values for label displayed on the button and for the label displayed in the filter; it is also possible to use a single configuration for all the dashboards, as the visualization will display only buttons relevant to the currently displayed dashboard.

Click on the Dashboard tab to go back to the Companies dashboard.

Note
for more informations about relational filters, please read the Relational filter chapter.

Query based aggregations

It is possible to get additional information about companies by using the results of queries on SQL databases (or any of the datasources supported by Kibi) as aggregations on Elasticsearch documents.

For example, in the Query on Companies visualization you can see that 40 of the 96 companies have competitors and 11 of them are in the top 500 companies by number of employees:

SQL based aggregations

Companies "With competitors" and Top 500 companies (HR count) are queries on the SQLite database; the records returned by the queries are used to filter Elasticsearch documents, which can be then aggregated in a metric.

To better understand this feature, let’s have a look at the Top 500 companies (HR count) query; to see the query, click on the Settings tab, then on Queries and on the Open button:

The query editor

The query returns the id, label and number_of_employees columns from the company table for the top 500 companies by number of employees:

select id, label, number_of_employees
from company
where number_of_employees>0
order by number_of_employees desc
limit 500

Click on the Dashboard tab, then click on the pencil icon in the heading of the Query on Companies visualization to customize its configuration:

Editing the Query on Companies visualization

The metrics section defines the aggregations on Elasticsearch documents, displayed as columns in the table; the buckets section defines the groups of Elasticsearch documents aggregated by metrics, displayed as row headers in the table.

By expanding the Split Rows section inside buckets you can see how the queries are used to define groups of Elasticsearch documents:

Query on Companies configuration

Scroll down to see the configuration of the fourth filter:

Configuration of an external query terms filter

The filter is configured to execute the query Top 500 companies (HR count) on the SQLite database and return the group of Elasticsearch documents from the current search whose id is equal to one of the id’s in the query results; the documents are then processed by the Count metric.

Let’s add a new aggregation to show the average number of employees; click on Add metrics inside the metrics section, then select Metric as the metric type; select Average as the aggregation and number_of_employees as the field, the click on the green button to apply changes.

Save the visualization by clicking on the Save button, confirm that you want to overwrite the existing visualization, then click on the Dashboard tab to see the updated visualization in the Companies dashboard:

Average aggregation

Click Add sub-buckets at the bottom, then select Split Slices. Choose the Terms aggregation and the age field from the drop-downs. Click the green Apply changes button apply-changes-button to add an external ring with the new results.

Note
read the Aggregation Builder chapter for an in-depth explanation of aggregations.

Besides defining groups to aggregate, queries can be used as filters; click on the Dashboard tab, then click on the Top-500-companies-(HR-count) row to see only the 11 companies mentioned in the articles which are also in the top 500 by number of employees:

Filter dashboard using a SQL query

Datasource entity selection

It is possible to select a company entity (record) in the SQLite database ( and entities in external datasources in general) by clicking on its label in the Companies Table.

The selected entity can be used as a parameter in queries; for example, click on Baidu in Companies Table:

Entity selection

Selecting an entity enables additional queries on external datasources; for example, in the Query on Companies visualization you can see that, amongst the top 500 companies by number of employees mentioned in articles about wireless or wifi, Baidu has one competitor and there are five companies in the same domain. All widgets affected by the selected entity are marked by a purple header.

For the Y-axis metrics aggregation, select Unique Count, with speaker as the field. For Shakespeare plays, it might be useful to know which plays have the lowest number of distinct speaking parts, if your theater company is short on actors. For the X-Axis buckets, select the Terms aggregation with the play_name field. For the Order, select Ascending, leaving the Size at 5.

Leave the other elements at their default values and click the green Apply changes button apply-changes-button. Your chart should now look like this:

Selecting an entity also enables the display of additional data in the Company Info visualization; by clicking on the (show) links you can toggle the list of companies in the same domain and competitors; the data in the tables is fetched from queries on the SQLite database, using the selected company ID as a parameter. The queries are rendered using customizable templates, which will be introduced later.

The selected entity appears as a purple box on the right of the filter bar; to deselect an entity, click on the bin icon displayed when moving the mouse over the purple box.

Note
for additional documentation about entity selection, please read the Datasource entity selection section in the External datasources chapter.

Enhanced search results

The Enhanced search results visualization displays the current set of Elasticsearch documents as a table; for example, Companies Table is configured to display the following fields:

  • Time (foundation date)

  • label (the company name)

  • description

  • category_code

  • founded_year

  • countrycode

  • Why Relevant? (a relational column)

Companies table

By clicking on the pencil icon, you can choose which fields to display and customize the order of the columns; if the index is time based, the Time column will be always displayed.

For example, expand the first row by clicking on the right arrow, then scroll down to the homepage_url field and click on the Toggle column icon:

Companies table preview

Click on the arrows to move the column to the desired position:

Column positioning

Click handlers

You can define click handlers on cells to perform several actions; let’s add a click handler to open the company homepage when clicking on the cell displaying the URL.

The table is pre-configured with a click handler on label that is used to select an entity in the SQLite database.

To add a new click handler, scroll down view options and click on Add click handler; select homepage_url in the Column dropdown, then Follow the URL in the On click I want to dropdown. Select homepage_url as the URL field, then click on the green button to apply changes.

You can test the click handler immediately by clicking on a cell displaying an homepage URL in the preview displayed on the right:

URL click handler

Relational column

The relational column can be enabled to display if an Elasticsearch document is matched by a query on the SQLite database.

For example, in the Companies Table, you can see that Verizon is in the top 50 companies by number of employees by looking at the Why Relevant? column because the label-not-analyzed field of the corresponding Elasticsearch document is matched by the label column in at least one of the records returned by the Top 50 companies (HR count) query.

Queries set in the relational column configuration can also take the selected entity as a parameter, so you can see that Yahoo! is both a competitor and a company in the same domain as Baidu:

Relational column example
Relational column configuration

Saving the visualization

Click on the save button in the top right to save the visualization, then click on the Dashboard tab to go back to the Companies dashboard.

Note
for additional documentation about this visualization, please read the Enhanced search results chapter.

Query templates

Company Info, which is an instance of a Kibi query viewer visualization, displays the results of three SQL queries by rendering their results through templates; the queries take the selected entity ID as an input, thus the associated templates will be displayed only when an entity is selected.

Kibi query viewer example

The association between query and templates can be set in the visualization configuration:

Kibi query viewer configuration

Query templates can be managed by clicking on the Settings tab, then on the Query templates tab.

Note
you can find the documentation about templates in the External datasources chapter; the visualization is documented in the Kibi query viewer chapter.

Loading data into Elasticsearch

This chapter contains basic information on how to load data into Elasticsearch for evaluation purposes.

From a SQL database using Logstash

The indices in the Kibi demo distribution have been populated by running four Logstash configurations over the SQLite database in kibi/crunchbase.db.

The database has the following schema:

SQLite database schema

Index setup

Before loading data, we need to setup indices and mappings; for example, let’s create an index called company-minimal in the Elasticsearch cluster at http://localhost:9220.

Create the index by running the following command in a terminal window:

curl -X PUT http://localhost:9220/company-minimal

If curl is not available on your system, please download it from http://curl.haxx.se/download.html .

If the index is created correctly, Elasticsearch will return the following response:

{"acknowledged":true}

If you want to destroy the index and start from scratch, execute the following command:

curl -X DELETE http://localhost:9220/company-minimal

Mapping definition

Mappings allow to configure how documents are stored in the index; for example, they allow to define how fields are matched by the search engine and set their type (string, dates, numbers, locations etc.).

Note
for detailed documentation about indices and mappings we recommend reading the Elasticsearch Reference.

Let’s define a simple mapping to describe a company; the mapping will define the following fields:

  • id: the id of the company in the SQLite database

  • name: the name of the company

  • description: a description of the company

  • homepage: the URL of the company homepage

  • number_of_employees: the number of employees

  • location: the geographical coordinates of the company

Open a text editor and paste the following text:

{
    "CompanyMinimal": {
        "properties": {
            "id": {
                "index": "not_analyzed",
                "type": "string"
            },
            "number_of_employees": {
                "type": "long"
            },
            "name": {
                "index": "analyzed",
                "type": "string"
            },
            "description": {
                "index": "analyzed",
                "type": "string"
            },
            "homepage": {
                "index": "not_analyzed",
                "type": "string"
            },
            "location": {
                "geohash": true,
                "type": "geo_point"
            }
        }
    }
}

CompanyMinimal is the name of the mapping; properties contains the indexing options for each field.

The type attribute specifies the field type; the example mapping contains four strings, a number and a geo_point, which allows to perform geographically aware search queries.

The index attribute in each field definition is used to control how the field will be handled by search engine; if set to not_analyzed, the search engine will match a document only if its field value is equal to the value set in the search query, if set to analyzed the field value will be processed by an analyzer to allow full-text queries.

It is also possible to set the index attribute to no to make the search engine ignore the field.

Save the file to demo/example/CompanyMinimal.mapping inside the directory where you extracted the demo distribution.

To apply the mapping, execute the following command:

curl -X PUT "http://localhost:9220/company-minimal/_mapping/CompanyMinimal" -d "@demo/example/CompanyMinimal.mapping"

If the mapping is created correctly, Elasticsearch will return the following response:

{"acknowledged":true}

SQL query definition

To extract the values that will be loaded to the index by Logstash, we need to write a SQL query; open a text editor and paste the following one:

SELECT id,
  label AS name,
  description,
  homepage_url as homepage,
  number_of_employees,
  CASE WHEN lat IS NULL THEN
    NULL
  ELSE
    lat || ', ' || lng
  END AS location
  FROM company
  LEFT JOIN company_geolocation ON company.id = company_geolocation.companyid

Save the file to demo/example/company-minimal.sql inside the directory where you extracted the demo distribution.

Logstash configuration

We now need to write a Logstash configuration to process the records returned by the query and populate the company-minimal index.

Note
Support for SQL databases is provided by the Logstash jdbc input plugin; User must download logstash to demo/example directory and install the required plugin

Open a text editor and paste the following:

input {
  jdbc {
    jdbc_driver_library => "sqlitejdbc-v056.jar"
    jdbc_driver_class => "org.sqlite.JDBC"
    jdbc_connection_string => "jdbc:sqlite:crunchbase.db"
    jdbc_user => ""
    jdbc_password => ""
    statement_filepath => "company-minimal.sql"
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
  }
}

filter {
  mutate {
    remove_field => ["@timestamp", "@version"]
  }
}

output {
  elasticsearch {
    host => "localhost"
    protocol => "http"
    port => 9220
    manage_template => false
    action => "index"
    index => "company-minimal"
    document_type => "CompanyMinimal"
  }
}

The statement_filepath parameter specifies the path to the file containing the SQL query; the jdbc_* parameters set the database connection string and authentication options.

The mutate filter is configured to remove default Logstash fields which are not needed in the destination index.

The output section specifies the destination index; manage_template is set to false as the index mapping has been explicitly defined in the previous steps.

Save the file to demo/example/company-minimal.conf

Copy the SQLite database to demo/example/crunchbase.db, then go to the demo/example directory and run the following command:

cd demo/example
logstash/bin/logstash -f company-minimal.conf

Logstash will execute the query and populate the index.

Note
for more information about Logstash, we recommend reading the Logstash reference and the jdbc input plugin documentation.

Browsing the index in Kibi

Open http://localhost:5606 or for installation with Shield open https://localhost:5606 in your browser, click on the Settings tab then on Indices .

Deselect Index contains time-based events, then write company-minimal in the Index name or pattern field:

Adding the company-minimal index

Click on Create to create the the index reference, then click on the Discover tab and select company-minimal in the dark grey dropdown:

Discovering the company-minimal index

Click on the right arrow at the beginning of each row to expand it and see all the loaded fields:

Viewing all the fields in a document

Demo distribution data loading script

The complete demo data loading process can be repeatead by running the demo/sql/bin/index_crunchbase_sqlite.sh script; the script performs the following actions:

  • Creates a copy of the database in the directory containing Logstash configurations

  • Creates the indices article, company, investor and investment

  • Sets the mappings for each index

  • Runs the logstash configuration for each index

The Logstash configurations and Elasticsearch mappings are available in the demo/sql/crunchbase/conf/logstash_sqlite directory.

Kibi Plugins added

Add-on functionality for Kibi is implemented with plug-in modules. You can use the bin/kibi plugin command to manage these modules. You can also install a plugin manually by moving the plugin file to the installedPlugins directory and unpacking the plugin files into a new directory.

A list of existing Kibana plugins is available on GitHub.

Installing Plugins

Use the following command to install a plugin:

bin/kibi plugin --install <org>/<package>/<version>
Note

Elasticsearch plugins follow the elasticsearch versionning, e.g. to installing Marvel plugin for this perticular Kibi version which comes by default with elasticsearch v2.4.4 use:

bin/kibi plugin -i elasticsearch/marvel/2.4.4

You can also use -i instead of --install, as in the following example:

bin/kibi plugin -i elasticsearch/marvel/latest

Because the organization given is elasticsearch, the plugin management tool automatically downloads the plugin from download.elastic.co.

Installing Plugins from an Arbitrary URL

You can specify a URL to a plugin with the -u or --url options after the -i or --install option, as in the following example:

bin/kibi plugin -i sample-plugin -u https://some.sample.url/directory
Installing sample-plugin
Attempting to extract from https://some.sample.url/directory
Downloading <some number> bytes....................
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

You can specify URLs that use the HTTP, HTTPS, or file protocols.

Installing Plugins to an Arbitrary Directory

Use the -d or --plugin-dir option to specify a directory for plugins, as in the following example:

bin/kibi plugin -i elasticsearch/sample-plugin/latest -d <path/to/directory>
Installing sample-plugin
Attempting to extract from https://download.elastic.co/elasticsearch/sample-plugin/sample-plugin-latest.tar.gz
Downloading <some number> bytes....................
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete
Note
This command creates the specified directory if it does not already exist.

Removing Plugins

Use the --remove or -r option to remove a plugin, including any configuration information, as in the following example:

bin/kibi plugin --remove marvel

You can also remove a plugin manually by deleting the plugin’s subdirectory under the installedPlugins directory.

Listing Installed Plugins

Use --list or -l option to list the currently installed plugins.

Updating Plugins

To update a plugin, remove the current version and reinstall the plugin.

Configuring the Plugin Manager

By default, the plugin manager provides you with feedback on the status of the activity you’ve asked the plugin manager to perform. You can control the level of feedback with the --quiet and --silent options. Use the --quiet option to suppress all non-error output. Use the --silent option to suppress all output.

By default, plugin manager requests do not time out. Use the --timeout option, followed by a time, to change this behavior, as in the following examples:

Waits for 30 seconds before failing
bin/kibi plugin --install username/sample-plugin --timeout 30s
Waits for 1 minute before failing
bin/kibi plugin --install username/sample-plugin --timeout 1m

Plugins and Custom Kibi Configurations

Use the -c or --config options to specify the path to the configuration file used to start Kibi. By default, Kibi uses the configuration file config/kibi.yml. When you change your installed plugins, the bin/kibi plugin command restarts the Kibi server. When you are using a customized configuration file, you must specify the path to that configuration file each time you use the bin/kibi plugin command.

Plugin Manager Exit Codes

0

Success

64

Unknown command or incorrect option parameter

74

I/O error

70

Other error

Switching Plugin Functionality

The Kibi UI serves as a framework that can contain several different plugins. You can switch between these plugins by clicking the Plugin Chooser Plugin chooser button to display icons for the installed plugins:

app picker

Click a plugin’s icon to switch to that plugin’s functionality.

Known Plugins for Kibana 4.x

Important
Plugin compatibility

The Kibana plugin interfaces are in a state of constant development. We cannot provide backwards compatibility for plugins due to the high rate of change. Kibana enforces that the installed plugins match the version of Kibana itself. Plugin developers will have to release a new version of their plugin for each new Kibana release as a result.

This list of plugins is not guaranteed to work on your version of Kibana. Instead, these are plugins that were known to work at some point with Kibana 4.x.

Apps

  • Sense - A JSON aware developer’s interface to Elasticsearch. Comes with handy machinery such as syntax highlighting, autocomplete, formatting and code folding

  • Timelion - Time series composer for Elasticsearch and beyond

  • LogTrail - View, analyze, search and tail log events in realtime with a developer/sysadmin friendly interface

Timelion Extensions

  • Mathlion (fermiumlabs) - enables equation parsing and advanced math under Timelion

Visualizations

Other

Note
If you want your plugin to be added to this page, open a pull request.

Accessing Kibi

Kibi is a web application that you access through port 5606. All you need to do is point your web browser at the machine where Kibana is running and specify the port number. For example, http://localhost:5606 or http://YOURDOMAIN.com:5606.

When you access Kibi, the Discover page loads by default with the default index pattern selected. The time filter is set to the last 15 minutes and the search query is set to match-all (\*).

If you don’t see any documents, try setting the time filter to a wider time range. If you still don’t see any results, it’s possible that you don’t have any documents.

Checking Kibi Status

You can reach the Kibi server’s status page by navigating to http://localhost:5606/status. The status page displays information about the server’s resource usage and lists the installed plugins.

kibi status page

Collecting Elasticsearch diagnostics

The Elasticsearch diagnostics button generates a single file by collecting different metrics about your elasticsearch cluster. All collected information are saved to a local file and never transfered over a network. User can see a full list of elasticsearch API calls by clicking the more info icon next to the button.

kibi status page diagnostics help

Discover

You can interactively explore your data from the Discover page. You have access to every document in every index that matches the selected index pattern. You can submit search queries, filter the search results, and view document data. You can also see the number of documents that match the search query and get field value statistics. If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in a histogram at the top of the page.

Note
As the discover functionality has not been modified by Kibi, we quote here the original Kibana documentation.

Discover Page

Setting the Time Filter

The Time Filter restricts the search results to a specific time period. You can set a time filter if your index contains time-based events and a time-field is configured for the selected index pattern.

By default the time filter is set to the last 15 minutes. You can use the Time Picker to change the time filter or select a specific time interval or time range in the histogram at the top of the page.

To set a time filter with the Time Picker:

  1. Click the Time Filter displayed in the upper right corner of the menu bar to open the Time Picker.

  2. To set a quick filter, simply click one of the shortcut links.

  3. To specify a relative Time Filter, click Relative and enter the relative start time. You can specify the relative start time as any number of seconds, minutes, hours, days, months, or years ago.

  4. To specify an absolute Time Filter, click Absolute and enter the start date in the From field and the end date in the To field.

  5. Click Apply button to set the selected time range to only the current dashboard or select other dashboards from the list and then click Apply button to set selected time range to more than one dashboard.

  6. Click the caret at the bottom of the Time Picker to hide it.

To set a Time Filter from the histogram, do one of the following:

  • Click the bar that represents the time interval you want to zoom in on.

  • Click and drag to view a specific timespan. You must start the selection with the cursor over the background of the chart—​the cursor changes to a plus sign when you hover over a valid start point.

You can use the browser Back button to undo your changes.

The histogram lists the time range you’re currently exploring, as well as the intervals that range is currently using. To change the intervals, click the link and select an interval from the drop-down. The default behavior automatically sets an interval based on the time range.

You can search the indices that match the current index pattern by submitting a search from the Discover page. You can enter simple query strings, use the Lucene query syntax, or use the full JSON-based Elasticsearch Query DSL.

When you submit a search, the histogram, Documents table, and Fields list are updated to reflect the search results. The total number of hits (matching documents) is shown in the upper right corner of the histogram. The Documents table shows the first five hundred hits. By default, the hits are listed in reverse chronological order, with the newest documents shown first. You can reverse the sort order by by clicking on the Time column header. You can also sort the table using the values in any indexed field. For more information, see Sorting the Documents Table.

To search your data:

  1. Enter a query string in the Search field:

    • To perform a free text search, simply enter a text string. For example, if you’re searching web server logs, you could enter safari to search all fields for the term safari.

    • To search for a value in a specific field, you prefix the value with the name of the field. For example, you could enter status:200 to limit the results to entries that contain the value 200 in the status field.

    • To search for a range of values, you can use the bracketed range syntax, [START_VALUE TO END_VALUE]. For example, to find entries that have 4xx status codes, you could enter status:[400 TO 499].

    • To specify more complex search criteria, you can use the Boolean operators AND, OR, and NOT. For example, to find entries that have 4xx status codes and have an extension of php or html, you could enter status:[400 TO 499] AND (extension:php OR extension:html).

      Note
      These examples use the Lucene query syntax. You can also submit queries using the Elasticsearch Query DSL. For examples, see query string syntax in the Elasticsearch Reference.
  2. Press Enter or click the Search button to submit your search query.

To clear the current search and start a new search, click the New Search button in the Discover toolbar.

New Search

You can reload saved searches on the Discover page and use them as the basis of visualizations. Saving a search saves both the search query string and the currently selected index pattern.

To save the current search:

  1. Click the Save Search button Save Search button in the Discover toolbar.

  2. Enter a name for the search and click Save.

To load a saved search:

  1. Click the Load Search button Load Search button in the Discover toolbar.

  2. Select the search you want to load.

If the saved search is associated with a different index pattern than is currently selected, loading the saved search also changes the selected index pattern.

Changing Which Indices You’re Searching

When you submit a search request, the indices that match the currently-selected index pattern are searched. The current index pattern is shown below the search field. To change which indices you are searching, click the name of the current index pattern to display a list of the configured index patterns and select a different index pattern.

For more information about index patterns, see Creating an Index Pattern.

Automatically Refreshing the Page

You can configure a refresh interval to automatically refresh the page with the latest index data. This periodically resubmits the search query.

When a refresh interval is set, it is displayed to the left of the Time Filter in the menu bar.

To set the refresh interval:

  1. Click the Time Filter Time Filter in the upper right corner of the menu bar.

  2. Click the Refresh Interval tab.

  3. Choose a refresh interval from the list.

To automatically refresh the data, click the autorefresh Auto-refresh button and select an autorefresh interval:

autorefresh intervals

When auto-refresh is enabled, Kibi’s top bar displays a pause button and the auto-refresh interval: autorefresh-pause. Click the Pause button to pause auto-refresh.

Filtering by Field

You can filter the search results to display only those documents that contain a particular value in a field. You can also create negative filters that exclude documents that contain the specified field value.

You can add filters from the Fields list or from the Documents table. When you add a filter, it is displayed in the filter bar below the search query. From the filter bar, you can enable or disable a filter, invert the filter (change it from a positive filter to a negative filter and vice-versa), toggle the filter on or off, or remove it entirely. Click the small left-facing arrow to the right of the index pattern selection drop-down to collapse the Fields list.

To add a filter from the Fields list:

  1. Click the name of the field you want to filter on. This displays the top five values for that field. To the right of each value, there are two magnifying glass buttons—​one for adding a regular (positive) filter, and one for adding a negative filter.

  2. To add a positive filter, click the Positive Filter button Positive Filter Button. This filters out documents that don’t contain that value in the field.

  3. To add a negative filter, click the Negative Filter button Negative Filter Button. This excludes documents that contain that value in the field.

To add a filter from the Documents table:

  1. Expand a document in the Documents table by clicking the Expand button Expand Button to the left of the document’s entry in the first column (the first column is usually Time). To the right of each field name, there are two magnifying glass buttons—​one for adding a regular (positive) filter, and one for adding a negative filter.

  2. To add a positive filter based on the document’s value in a field, click the Positive Filter button Positive Filter Button. This filters out documents that don’t contain the specified value in that field.

  3. To add a negative filter based on the document’s value in a field, click the Negative Filter button Negative Filter Button. This excludes documents that contain the specified value in that field.

Working with Filters

When you create a filter anywhere in Kibi, the filter conditions display in a green oval under the search text entry box:

filter sample

Hovering on the filter oval displays the following icons:

filter allbuttons
Enable Filter filter-enable

Click this icon to disable the filter without removing it. You can enable the filter again later by clicking the icon again. Disabled filters display a striped shaded color, green for inclusion filters and red for exclusion filters.

Pin Filter filter-pin

Click this icon to pin a filter. Pinned filters persist across Kibi tabs. You can pin filters from the Visualize tab, click on the Discover or Dashboard tabs, and those filters remain in place.

Note
If you have a pinned filter and you’re not seeing any query results, check that your current tab’s index pattern is one that the filter applies to. E.g. a filter "name:giovanni" will results in 0 results if pinned and therefore "dragged along" to a dashboard whose underlying index does not have a "name" field, let alone a "giovanni" value. For this reason a good pattern in Kibi is to use Dashboard Groups to group together dashboard which are based on the same underlying index. In this case the user can safely pin and "drag along" a filter across dashboards in the same group.
Toggle Filter filter-toggle

Click this icon to toggle a filter. By default, filters are inclusion filters, and display in green. Only elements that match the filter are displayed. To change this to an exclusion filters, displaying only elements that don’t match, toggle the filter. Exclusion filters display in red.

Remove Filter filter-delete

Click this icon to remove a filter entirely.

Custom Filter filter-custom

Click this icon to display a text field where you can customize the JSON representation of the filter and specify an alias to use for the filter name:

filter custom json

You can use JSON filter representation to implement predicate logic, with should for OR, must for AND, and must_not for NOT:

Example 1. OR Example
{
  "bool": {
    "should": [
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "China"
        }
      }
    ]
  }
}
Example 2. AND Example
{
  "bool": {
    "must": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.city_name.raw": "New York"
        }
      }
    ]
  }
}
Example 3. NOT Example
{
  "bool": {
    "must_not": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      }
    ]
  }
}

Click the Done button to update the filter with your changes.

To apply any of the filter actions to all the filters currently in place, click the filter-actions Global Filter Actions button and select an action.

Viewing Document Data

When you submit a search query, the 500 most recent documents that match the query are listed in the Documents table. You can configure the number of documents shown in the table by setting the discover:sampleSize property in Advanced Settings. By default, the table shows the localized version of the time field specified in the selected index pattern and the document _source. You can add fields to the Documents table from the Fields list. You can sort the listed documents by any indexed field that’s included in the table.

To view a document’s field data, click the Expand button Expand Button to the left of the document’s entry in the first column (the first column is usually Time). Kibi reads the document data from Elasticsearch and displays the document fields in a table. The table contains a row for each field that contains the name of the field, add filter buttons, and the field value.

Expanded Document
  1. To view the original JSON document (pretty-printed), click the JSON tab.

  2. To view the document data as a separate page, click the link. You can bookmark and share this link to provide direct access to a particular document.

  3. To collapse the document details, click the Collapse button Collapse Button.

  4. To toggle a particular field’s column in the Documents table, click the Add Column Toggle column in table button.

Sorting the Document List

You can sort the documents in the Documents table by the values in any indexed field. If a time field is configured for the selected index pattern, by default the documents are sorted in reverse chronological order.

To change the sort order:

  • Click the name of the field you want to sort by. The fields you can use for sorting have a sort button to the right of the field name. Clicking the field name a second time reverses the sort order.

Adding Field Columns to the Documents Table

By default, the Documents table shows the localized version of the time field specified in the selected index pattern and the document _source. You can add fields to the table from the Fields list or from a document’s expanded view.

To add field columns to the Documents table:

  1. Mouse over a field in the Fields list and click its add button Add Field Button.

  2. Repeat until you’ve added all the fields you want to display in the Documents table.

  3. Alternately, add a field column directly from a document’s expanded view by clicking the Add Column Toggle column in table button.

The added field columns replace the _source column in the Documents table. The added fields are also listed in the Selected Fields section at the top of the field list.

To rearrange the field columns in the table, mouse over the header of the column you want to move and click the Move button.

Move Column

Removing Field Columns from the Documents Table

To remove field columns from the Documents table:

  1. Mouse over the field you want to remove in the Selected Fields section of the Fields list and click its remove button Remove Field Button.

  2. Repeat until you’ve removed all the fields you want to drop from the Documents table.

Viewing Field Data Statistics

From the field list, you can see how many documents in the Documents table contain a particular field, what the top 5 values are, and what percentage of documents contain each value.

To view field data statistics:

  • Click the name of a field in the Fields list. The field can be anywhere in the Fields list—​Selected Fields, Popular Fields, or the list of other fields.

Field Statistics

Tip
To create a visualization based on the field, click the Visualize button below the field statistics.

External datasources

Kibi provides visualizations and aggregations to integrate data from external datasources; this section explains how to setup datasources and configure queries and query templates.

Configuration

To create a new external datasource navigate to "Setttings/Datasources" First fill datasource title, name and pick the type Kibi supports following datasource types:

  • REST

  • SQLite

  • MySQL

  • PostgreSQL

  • Sparql over http - any SPARQL HTTP endpoint (tested with Openlink Virtuoso and Joseki)

  • SQL JDBC - experimental JDBC support for SQL (tested with H2).

  • SPARQL JDBC - experimental JDBC support for SPARQL

  • Kibi Tinkerpop3 [Enterprise Edition only] - experimental support for Kibi Gremlin Server

After selecting the datasource type additional parameters will appear. Below list of additional mandatory parameters for each datasource type:

Common parameters

  • timeout - connection timeout in milliseconds

  • cache_enabled - enable server side cache for this datasource

  • max_age - the max age of an object in the cache, in milliseconds

To control the maximum number of query results kept in cache, set the kibi_core.datasource_cache_size parameter in kibi.yml.

Note
A Kibi restart is required to apply changes to kibi.yml configuration file.

REST

  • url - an URL of rest API endpoint

  • response_type - API results format, currently Kibi supports only JSON

  • username (optional) - username with access to the API

  • password (optional) - corresponding password

  • auth_token (optional) - token used in Token Authentication

SQLite

  • db_file_path - path to database file

MySQL, PostgreSQL

  • host - hostname

  • dbname - database name

  • username - username with read access to the database

  • password - corresponding password

Sparql over http

  • endpoint_url - url of the SPARQL http endpoint

SQL JDBC, SPARQL JDBC

  • connection_string - JDBC connection string without username and password e.g.: "jdbc:postgresql://localhost/mydatabase"

  • libpath - path to folder with your jars e.g.: "/otp/dbdrivers"

  • drivername - e.g.: "org.postgresql.Driver"

  • libs - comma separated list of jars (they can be just filenames then it is asumend they are in libpath folder or they can have absolute paths to jars)

Parameters encryption

Sensitive datasource parameters like passwords are encrypted before being stored in the .kibi index.

Before creating datasources containing sensitive parameters, make sure to set a custom encryption key by running the replace_encryption_key command:

bin/kibi replace_encryption_key [options] <current_key> <new_key> <new_cipher>
  • <current_key> a base64 encoded string containing the current encryption key.

  • <new_key> a base64 encoded string containing the new encryption key.

  • <new_cipher> the cipher algorithm to use (currently only AES-GCM is supported).

The current encryption key can be read from the kibi.yml file in the datasource_encryption_key parameter.

Keys can have a length of 16, 24 or 32 bytes; a quick way to encode a plaintext string to base64 is to use the base64 utility from the coreutils package:

$ echo -n changemechangemechangemechangeme | base64
Y2hhbmdlbWVjaGFuZ2VtZWNoYW5nZW1lY2hhbmdlbWU=
Note
Make sure to set the configuration file as readable only by the user running the Kibi process.

Datasource entity selection

Selected Entities can be used to as source of parameters for queries Each selected entity is uniquely identified by an URI:

  • INDEX/TYPE/ID where INDEX is an index pattern, TYPE is a type of a document, and ID is document ID

As explained in the following sections, queries on external datasources can extract variables from the selected entity URI; in order to allow the user to select an entity, you must add an Enhanced search results visualization to a dashboard and configure at least one click handler to select an entity.

Once the visualization is configured, clicking on the cell will display a purple box in the filter bar, and the variables stored in the entity URI will be available to queries and query templates.

The screenshot below shows the effect of clicking on a cell configured with an entity selection handler; after selecting an entity, the Company Info template viewer shows the information about the company fetched by a query.

Entity selection
Entity selection configuration example

To disable or cancel the selection, click on the icons displayed inside the entity selection widget when the mouse is over it, as displayed in the screenshot below:

Entity selection options

Query templates

You can define templates to format query results.

Kibi supports three template engines:

There are four pre-defined templates:

  • kibi-html-angular: this template for each document displays a panel populated with all property values (Currently supported only in Enhanced search results visualisation)

  • kibi-json-jade: this template presents the query results as a pretty-printed JSON object using the jade engine. This is useful to test queries while writing them.

  • kibi-table-jade: this template displays the query results in a table, using the jade engine.

  • kibi-table-handlebars: like kibi-table-jade, using the handlebars engine instead.

You can define your own custom template by clicking on the Settings / Templates tab.

Then, pick the engine you prefer and write the template; to see a preview, click on the save button and select a query from the list; depending on the query you selected, the EntityURI may need to be set.

Query template editor

Queries

Queries can be used to provide data to Query templates, tag and filter Elasticsearch documents.

To create a new query, click to the "Settings/Queries" tab.

You need then to set the following fields to define a query:

  • Title: the title of the query.

  • Datasource: the name of a configured datasource.

  • Results query: the query declaration.

You may also set a description for the query and one or more tags.

Below is an example configuration of a query on a SQL database called Top 50 companies (HR count) that returns the Top 50 companies by number of employees in a table called company.

Configuration of a SQL endpoint

The preview section will display the results of the query as a table or as a JSON object.

Note
Template rendering is currently a blocking operation, therefore queries returning a large number of results might make the backend unresponsive for an indeterminate amount of time.

Query variables:

One of the most useful features of queries is that it is possible to set some of their parameters before execution by using datasource specific variables, which can be set at runtime by configuring click handlers in the Enhanced search results visualization to select an entity.

Variable values are taken from elasticsearch document selected via selected entity URI.

All properties from selected document can be accessed using the following syntax: @doc[PATH_ELEMENT_1][PATH_ELEMENT_2]…​[PATH_ELEMENT_N]@

  • to get the document id use: @doc[_id]@

  • to get the value of property called action use: @doc[_source][action]@

  • to get the value of nested property called person.age use: @doc[_source][person][age]@

In order to view the results of the query, you have to specify an entity URI manually in the field on the top right;

Below is an example of configuration for a query named Company Info using a variable to get the value of property called id of currently selected entity In the example, @doc[_source][id]@ is replaced with an id taken from selected company. In the Selected Entity box we see that the selected company is from index: company, has a type: Company and has the id AVgfaYQ0Q2VQXwxDgyfY

SQL query with variables

Activation Query

An activation query can be specified to conditionally execute the results query.

For example, if you have a table called Vehicles but some of the queries are only relevant to "Motorcycles" and not to "Cars", the activation query could be used to determine if the results query should be executed when an entity in Vehicles by looking at its type. If the query is not executed, any template or aggregator using the query will be automatically disabled.

On SQL datasources, activation queries will trigger results query execution when returning at least one record.

Example:

SELECT id
FROM Vehicles
WHERE id='@doc[_source][id]@' AND vehicle_type='Motorcycle'

On SPARQL datasources, activation queries must be written using the ASK form; the corresponding results query will be executed only if the query has a solution.

Example:

PREFIX vehicle: <http://ontologies.example.org/vehicle#>

ASK {
    <@doc[_source][uri]@> a vehicle:Motorcycle
}

Use cases

Once you’ve configured query templates and queries, you can use them in the following visualizations:

It is also possible to use queries as aggregations as explained below.

External query terms filters aggregation

The query results from an external data source can be used as an aggregation in visualizations.

This allows to compute metrics on Elasticsearch documents joined with query results.

To use a query as an aggregation, select a bucket type and select External Query Terms Filter in the Aggregation dropdown; then, click on the Add an external query terms filter button.

You can then configure how to join the query results with the Elasticsearch documents by setting the following parameters:

  • Source query id: the name of the query on the external datasource.

  • Source query variable: the name of the variable in query results which contains the first value used in the join.

  • Target field: the name of the field in the target index which contains the second value used in the join.

The aggregation will return only documents in the Elasticsearch index whose target field value is equal to the source query variable value in at least one of the results returned by the query; if Negate the query is checked, the aggregation will return only documents in the Elasticsearch index whose target field value is not equal to any of the values of the source query variable in the results returned by the query.

For example, the screenshot below show the configuration of a Data table visualization with three aggregations based on external queries:

  • A query that selects the labels of the competitors of the currently selected company

  • A query that selects the labels of all the companies which have a competitor

  • A query that selects the id’s of the top 500 companies by number of employees

If a query requires a selected entity, and no entity is selected, the computed aggregation will return 0, also the controls to select Selected entity will indicate (red borders arround) that it is necessary to select one.

Configuration of an external query terms filter aggregation on a data table visualization

The screenshot below shows the configuration of two external query terms filter aggregation on a pie chart visualization:

Configuration of an external query terms filter aggregation on a pie chart visualization

Kibi Gremlin Server [Enterprise Edition only]

NOTE: Documentation for Kibi Gremlin Server is available only in Kibi Enterprise Edition.

Sentinl

Sentinl extends Kibi/Kibana with Alerting and Reporting functionality to monitor, notify and report on data series changes using standard queries, programmable validators and a variety of configurable actions.

Sentinl is also designed to simplify the process of creating and managing alerts and reports in Kibi/Kibana via its integrated App and Spy integration.

Note
Public documentation for Sentinl is available here.

Join query

Join Query in Sentinl are available only in the Enterprise Edition.

Note
If you want to read more about join query, this is the documentation.

Visualize

Note
Visualization management has not been modified by Kibi; we quote the original Kibana documentation for all the standard data aggregations integrated with Kibi specific features were applicable.

You can use the Visualize page to design data visualizations. You can save these visualizations, use them individually, or combine visualizations into a dashboard. A visualization can be based on one of the following data source types:

  • A new interactive search

  • A saved search

  • An existing saved visualization

Visualizations are based on the aggregation feature introduced in Elasticsearch 1.x.

Creating a New Visualization

To start the New Visualization wizard, click on the Visualize tab at the top left of the page. If you are already creating a visualization, you can click the New Visualization button New Document button in the toolbar to the right of the search bar. The wizard guides you through the following steps:

Step 1: Choose the Visualization Type

Choose a visualization type when you start the New Visualization wizard:

Enhanced search results

Display an interactive table of documents returned by an Elasticsearch query.

Kibi Query Viewer

Display the results of queries on external datasources in Jade/Handlebards templates.

Relational Filter

Apply cross-dashboard filters based on the relations between different indices.

Area chart

Use area charts to visualize the total contribution of several different series.

Data table

Use data tables to display the raw data of a composed aggregation. You can display the data table for several other visualizations by clicking at the bottom of the visualization.

Line chart

Use line charts to compare different series.

Markdown widget

Use the Markdown widget to display free-form information or instructions about your dashboard.

Metric

Use the metric visualization to display a single number on your dashboard.

Pie chart

Use pie charts to display each source’s contribution to a total.

Tile map

Use tile maps to associate the results of an aggregation with geographic points.

Vertical bar chart

Use vertical bar charts as a general-purpose chart.

Kibi Word Cloud

Display a cloud of high frequency terms in a document collection.

Kibi Timeline

Displays series of data coming from different saved searches on a single timeline component.

Radar Chart

Display multivariate data with a two-dimensional chart.

Kibi Graph Browser*

Displays a graph visualization starting from a selected document

Kibi Bubble Diagram*

The Bubble Diagram visualization displays series of data grouped into packed circles.

Note
*: available only in Kibi Enterprise Edition.

You can also load a saved visualization that you created earlier. The saved visualization selector includes a text field to filter by visualization name and a link to the Object Editor, accessible through Settings > Edit Saved Objects, to manage your saved visualizations.

If your new visualization is a Markdown widget, selecting that type takes you to a text entry field where you enter the text to display in the widget. For all other types of visualization, selecting the type takes you to data source selection.

Step 2: Choose a Data Source

You can choose a new or saved search to serve as the data source for your visualization. Searches are associated with an index or a set of indices. When you select new search on a system with multiple indices configured, select an index pattern from the drop-down to bring up the visualization editor.

When you create a visualization from a saved search and save the visualization, the search is tied to the visualization. When you make changes to the search that is linked to the visualization, the visualization updates automatically.

Step 3: The Visualization Editor

The visualization editor enables you to configure and edit visualizations. The visualization editor has the following main elements:

VizEditor

Automatically Refreshing the Page

You can configure a refresh interval to automatically refresh the page with the latest index data. This periodically resubmits the search query.

When a refresh interval is set, it is displayed to the left of the Time Filter in the menu bar.

To set the refresh interval:

  1. Click the Time Filter Time Filter in the upper right corner of the menu bar.

  2. Click the Refresh Interval tab.

  3. Choose a refresh interval from the list.

To automatically refresh the data, click the autorefresh Auto-refresh button and select an autorefresh interval:

autorefresh intervals

When auto-refresh is enabled, Kibi’s top bar displays a pause button and the auto-refresh interval: autorefresh-pause. Click the Pause button to pause auto-refresh.

Toolbar

The toolbar has a search field for interactive data searches, as well as controls to manage saving and loading visualizations. For visualizations based on saved searches, the search bar is grayed out. To edit the search, replacing the saved search with the edited version, double-click the search field.

The toolbar at the right of the search box has buttons for creating new visualizations, saving the current visualization, loading an existing visualization, sharing or embedding the visualization, and refreshing the data for the current visualization.

Aggregation Builder

Use the aggregation builder on the left of the page to configure the metric and bucket aggregations used in your visualization. Buckets are analogous to SQL GROUP BY statements. For more information on aggregations, see the main Elasticsearch aggregations reference.

In addition to standard Elasticsearch aggregations, Kibi adds the External Query Terms Filter aggregation, which is described in the External datasources chapter.

Bar, line, or area chart visualizations use metrics for the y-axis and buckets are used for the x-axis, segment bar colors, and row/column splits. For pie charts, use the metric for the slice size and the bucket for the number of slices.

Choose the metric aggregation for your visualization’s Y axis, such as count, average, sum, min, max, or cardinality (unique count). Use bucket aggregations for the visualization’s X axis, color slices, and row/column splits. Common bucket aggregations include date histogram, range, terms, filters, and significant terms.

You can set the order in which buckets execute. In Elasticsearch, the first aggregation determines the data set for any subsequent aggregations. The following example involves a date bar chart of Web page hits for the top 5 file extensions.

To use the same extension across all hits, set this order:

  1. Color: Terms aggregation of extensions

  2. X-Axis: Date bar chart of @timestamp

Elasticsearch collects the records for the top 5 extensions, then creates a date bar chart for each extension.

To chart the top 5 extensions for each hour, use the following order:

  1. X-Axis: Date bar chart of @timestamp (with 1 hour interval)

  2. Color: Terms aggregation of extensions

For these requests, Elasticsearch creates a date bar chart from all the records, then groups the top five extensions inside each bucket, which in this example is a one-hour interval.

Note
Remember, each subsequent bucket slices the data from the previous bucket.

To render the visualization on the preview canvas, click the green Apply Changes button at the top right of the Aggregation Builder.

You can learn more about aggregation and how altering the order of aggregations affects your visualizations here.

Working with Filters

When you create a filter anywhere in Kibi, the filter conditions display in a green oval under the search text entry box:

filter sample

Hovering on the filter oval displays the following icons:

filter allbuttons
Enable Filter filter-enable

Click this icon to disable the filter without removing it. You can enable the filter again later by clicking the icon again. Disabled filters display a striped shaded color, green for inclusion filters and red for exclusion filters.

Pin Filter filter-pin

Click this icon to pin a filter. Pinned filters persist across Kibi tabs. You can pin filters from the Visualize tab, click on the Discover or Dashboard tabs, and those filters remain in place.

Note
If you have a pinned filter and you’re not seeing any query results, check that your current tab’s index pattern is one that the filter applies to. E.g. a filter "name:giovanni" will results in 0 results if pinned and therefore "dragged along" to a dashboard whose underlying index does not have a "name" field, let alone a "giovanni" value. For this reason a good pattern in Kibi is to use Dashboard Groups to group together dashboard which are based on the same underlying index. In this case the user can safely pin and "drag along" a filter across dashboards in the same group.
Toggle Filter filter-toggle

Click this icon to toggle a filter. By default, filters are inclusion filters, and display in green. Only elements that match the filter are displayed. To change this to an exclusion filters, displaying only elements that don’t match, toggle the filter. Exclusion filters display in red.

Remove Filter filter-delete

Click this icon to remove a filter entirely.

Custom Filter filter-custom

Click this icon to display a text field where you can customize the JSON representation of the filter and specify an alias to use for the filter name:

filter custom json

You can use JSON filter representation to implement predicate logic, with should for OR, must for AND, and must_not for NOT:

Example 4. OR Example
{
  "bool": {
    "should": [
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "China"
        }
      }
    ]
  }
}
Example 5. AND Example
{
  "bool": {
    "must": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.city_name.raw": "New York"
        }
      }
    ]
  }
}
Example 6. NOT Example
{
  "bool": {
    "must_not": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      }
    ]
  }
}

Click the Done button to update the filter with your changes.

To apply any of the filter actions to all the filters currently in place, click the filter-actions Global Filter Actions button and select an action.

Preview Canvas

The preview canvas displays a preview of the visualization you’ve defined in the aggregation builder. To refresh the visualization preview, clicking the Refresh button Refresh button on the toolbar.

Enhanced search results

Enhanced search results is a visualisation that shows the documents matched by a query on an Elasticsearch index, similar to the Kibana Discover table.

In addition to column configuration, the visualization provides the following features:

  • it is possible to enable a column that indicates whether or not a search result is matched by a query on an external datasource. This is described in Relational Column.

  • it is possible to define click handlers on the cells in a column, e.g. to open the URL displayed in a cell. This is described in Click handlers.

Configuration view of the Enhanced search results table

Relational Column

The relational column can be used to display if a search result is matched by a query on an external datasource.

To enable the relational column, check the Enable Relational Column checkbox.

The screenshot below shows the configuration of a relational column named Why Relevant? where the value of a cell depends on the query Top 50 companies (HR count): if the value of the label index field of a document matches the value of the label variable in at least one record returned by the query, the name of the query will be displayed inside the cell.

Relational column configuration
Relational column example

In order to configure the relational column, you must set the following parameters:

  • Column name: the column name that will be displayed in the table header.

  • Source Field: the name of the index field that will be compared to a variable in the query results.

  • Target query: the name of the query to execute.

  • Target query variable name: the name of the query variable that will be compared to the index field specified in Source field.

Click handlers

It is possible to define two different actions when clicking on a cell;

  • Open an URL defined in the corresponding index field.

  • Select an entity in an external datasource matching the corresponding index field.

Follow URL

Select the Follow URL action to open a URL stored in an index field in a new window.

For example, the following configuration defines an handler that opens the URL stored in the field homepage_url when clicking on the cell displaying the label field.

Follow URL on click

To configure a click handler, you must set the following parameters:

  • Column — the name of the column to which the handler will be bound.

  • On click I want to — the action to perform on click, select Follow the URL here.

  • URL field — the name of the field containing the URL.

  • URL format — a custom format string to compose the URL, where @URL@ is replaced with the value of the field set in URL field.

URL format can be used to create dynamic URL; the screenshot below shows a configuration in which the value of the the id field is used to define the path of an URL on example.org.

With this configuration, if the id field is set to 11 the resulting URL will be http://example.org/11 .

Follow URL with a custom format on click

Select an entity

Select the Select an entity action if you want to select an entity stored in an external datasource matching the selected Elasticsearch document; for more information about entity selection, please read the Datasource entity selection section.

To configure an entity selection action you must set the following parameters:

  • Column — the name of the column to which the handler will be bound.

  • On click I want to — the action to perform on click, select Select the document here.

  • Redirect to dashboard — if set, clicking on the cell will select the entity and display the specified dashboard.

Configuration of an entity selection handler

Kibi query viewer

This visualization displays the results from multiple queries on external data sources using query templates.

To add a query to the visualization, click on the Add query button, then set the following parameters:

  • Label: the caption for the table, in case of a table template like kibi-table-jade. This sets the variable label to the given value.

  • Source query: the query used by the template.

  • Template: the template used to render results returned by Source query.

If one of the source queries requires an entity to be selected, you can set an entity URI for testing in the input field above the preview.

If a source query is not activated, the corresponding template will not be rendered.

The screenshots below show the configuration and output of a Templated query viewer visualization for a selected company:

Configuration of a Kibi query viewer visualization

Advanced options

By clicking on the Advanced link, you can set additional rendering options.

It is possible to set additional template variables by writing them as JSON object properties in the Template variables textarea.

For example, to customize the heading of the generic table template (this is done automatically by the Label input field above), which is set by default to the id of the source query, you can customize the label variable as follows:

{
    "label": "Info"
}

By default, template contents are hidden and can be displayed by clicking on the show link in the heading; to make template contents visible by default, check Render opened box.

Advanced options

Area Charts

This chart’s Y axis is the metrics axis. The following aggregations are available for this axis:

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Percentiles

The percentile aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The percentile ranks aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

You can add an aggregation by clicking the + Add Aggregation button.

The X axis of this chart is the buckets axis. You can define buckets for the X axis, for a split area on the chart, or for split charts.

This chart’s X axis supports the following aggregations. Click the linked name of each aggregation to visit the main Elasticsearch documentation for that aggregation.

Date Histogram

A date histogram is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second.

Histogram

A standard histogram is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a range aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A date range aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using date math expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The IPv4 range aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A terms aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of filters for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the Label button icon label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental significant terms aggregation.

External query terms filter

a Kibi aggregator where one can define one or more buckets based on some record value (typically a primary key) matching the results of an external query. Multiple such buckets, corresponding to multiple queries, can be defined. For more information see the query menu in the configuration. This displays the results of the external query terms filter aggregation.

Once you’ve specified an X axis aggregation, you can define sub-aggregations to refine the visualization. Click + Add Sub Aggregation to define a sub-aggregation, then choose Split Area or Split Chart, then select a sub-aggregation from the list of types.

When multiple aggregations are defined on a chart’s axis, you can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

Enter a string in the Custom Label field to change the display label. For example, a chart of dates with incident counts can display dates in chronological order, or you can raise the priority of the incident-reporting aggregation to show the most active dates first. The chronological order might show a time-dependent pattern in incident count, and sorting by active dates can reveal particular outliers in your data.

You can customize the colors of your visualization by clicking the color dot next to each label to display the color picker.

An array of color dots that users can select

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Exclude Pattern Flags

A standard set of Java flags for the exclusion pattern.

Include Pattern

Specify a pattern in this field to include in the results.

Include Pattern Flags

A standard set of Java flags for the inclusion pattern.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the chart:

Chart Mode

When you have multiple Y-axis aggregations defined for your chart, you can use this drop-down to affect how the aggregations display on the chart:

stacked

Stacks the aggregations on top of each other.

overlap

The aggregations overlap, with translucency indicating areas of overlap.

wiggle

Displays the aggregations as a streamgraph.

percentage

Displays each aggregation as a proportion of the total.

silhouette

Displays each aggregation as variance from a central line.

Checkboxes are available to enable and disable the following behaviors:

Smooth Lines

Check this box to curve the top boundary of the area from point to point.

Set Y-Axis Extents

Check this box and enter values in the y-max and y-min fields to set the Y axis to specific values.

Scale Y-Axis to Data Bounds

The default Y axis bounds are zero and the maximum value returned in the data. Check this box to change both upper and lower bounds to match the values returned in the data.

Show Tooltip

Check this box to enable the display of tooltips.

Show Legend

Check this box to enable the display of a legend next to the chart.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Data Table

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The extended stats aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The percentile aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The percentile ranks aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

You can add an aggregation by clicking the + Add Aggregation button.

Enter a string in the Custom Label field to change the display label.

The rows of the data table are called buckets. You can define buckets to split the table into rows or to split the table into additional tables.

Each bucket type supports the following aggregations:

Date Histogram

A date histogram is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second.

Histogram

A standard histogram is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a range aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A date range aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using date math expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The IPv4 range aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A terms aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of filters for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the labelbutton label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental significant terms aggregation. The value of the Size parameter defines the number of entries this aggregation returns.

Geohash

The geohash aggregation displays points based on the geohash coordinates.

External query terms filter

this aggregation matches terms of a specified field against the value returned by an external query (SQL/REST etc). Multiple queries can be specified thus creating multiple aggregation bins. Queries are configured under Settings/Queries.

Once you’ve specified a bucket type aggregation, you can define sub-aggregations to refine the visualization. Click + Add Sub Aggregation to define a sub-aggregation, then choose Split Rows or Split Table, then select a sub-aggregation from the list of types.

You can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Exclude Pattern Flags

A standard set of Java flags for the exclusion pattern.

Include Pattern

Specify a pattern in this field to include in the results.

Include Pattern Flags

A standard set of Java flags for the inclusion pattern.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the table:

Per Page

This field controls the pagination of the table. The default value is ten rows per page.

Checkboxes are available to enable and disable the following behaviors:

Show metrics for every bucket/level

Check this box to display the intermediate results for each bucket aggregation.

Show partial rows

Check this box to display a row even when there is no result.

Note
Enabling these behaviors may have a substantial effect on performance.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Line Charts

This chart’s Y axis is the metrics axis. The following aggregations are available for this axis:

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The extended stats aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The percentile aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The percentile ranks aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

You can add an aggregation by clicking the + Add Aggregation button.

Enter a string in the Custom Label field to change the display label.

Before you choose a buckets aggregation, specify if you are splitting slices within a single chart or splitting into multiple charts. A multiple chart split must run before any other aggregations. When you split a chart, you can change if the splits are displayed in a row or a column by clicking the Rows | Columns selector.

The X axis of this chart is the buckets axis. You can define buckets for the X axis, for a split area on the chart, or for split charts.

This chart’s X axis supports the following aggregations. Click the linked name of each aggregation to visit the main Elasticsearch documentation for that aggregation.

Date Histogram

A date histogram is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second.

Histogram

A standard histogram is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a range aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A date range aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using date math expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The IPv4 range aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A terms aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of filters for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the Label button icon label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental significant terms aggregation.

External query terms filter

a Kibi aggregator where one can define one or more buckets based on some record value (typically a primary key) matching the results of an external query. Multiple such buckets, corresponding to multiple queries, can be defined. For more information see the query menu in the configuration. This displays the results of the external query terms filter aggregation.

Once you’ve specified an X axis aggregation, you can define sub-aggregations to refine the visualization. Click + Add Sub Aggregation to define a sub-aggregation, then choose Split Area or Split Chart, then select a sub-aggregation from the list of types.

When multiple aggregations are defined on a chart’s axis, you can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

Enter a string in the Custom Label field to change the display label.

You can customize the colors of your visualization by clicking the color dot next to each label to display the color picker.

An array of color dots that users can select

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Exclude Pattern Flags

A standard set of Java flags for the exclusion pattern.

Include Pattern

Specify a pattern in this field to include in the results.

Include Pattern Flags

A standard set of Java flags for the inclusion pattern.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the chart:

Y-Axis Scale

You can select linear, log, or square root scales for the chart’s Y axis. You can use a log scale to display data that varies exponentially, such as a compounding interest chart, or a square root scale to regularize the display of data sets with variabilities that are themselves highly variable. This kind of data, where the variability is itself variable over the domain being examined, is known as heteroscedastic data. For example, if a data set of height versus weight has a relatively narrow range of variability at the short end of height, but a wider range at the taller end, the data set is heteroscedastic.

Smooth Lines

Check this box to curve the line from point to point. Bear in mind that smoothed lines necessarily affect the representation of your data and create a potential for ambiguity.

Show Connecting Lines

Check this box to draw lines between the points on the chart.

Show Circles

Check this box to draw each data point on the chart as a small circle.

Current time marker

For charts of time-series data, check this box to draw a red line on the current time.

Set Y-Axis Extents

Check this box and enter values in the y-max and y-min fields to set the Y axis to specific values.

Show Tooltip

Check this box to enable the display of tooltips.

Show Legend

Check this box to enable the display of a legend next to the chart.

Scale Y-Axis to Data Bounds

The default Y-axis bounds are zero and the maximum value returned in the data. Check this box to change both upper and lower bounds to match the values returned in the data.

After changing options, click the green Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Bubble Charts

You can convert a line chart visualization to a bubble chart by performing the following steps:

  1. Click Add Metrics for the visualization’s Y axis, then select Dot Size.

  2. Select a metric aggregation from the drop-down list.

  3. In the Options tab, uncheck the Show Connecting Lines box.

  4. Click the Apply changes button.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Markdown Widget

The Markdown widget is a text entry field that accepts GitHub-flavored Markdown text. Kibi renders the text you enter in this field and displays the results on the dashboard. You can click the Help link to go to the help page for GitHub flavored Markdown. Click Apply to display the rendered text in the Preview pane or Discard to revert to a previous version.

Metric

A metric visualization displays a single number for each aggregation you select:

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The extended stats aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The percentile aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The percentile ranks aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

You can add an aggregation by clicking the + Add Aggregation button.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options:

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Click the Options tab to display the font size slider.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Pie Charts

The slice size of a pie chart is determined by the metrics aggregation. The following aggregations are available for this axis:

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, specify if you are splitting slices within a single chart or splitting into multiple charts. A multiple chart split must run before any other aggregations. When you split a chart, you can change if the splits are displayed in a row or a column by clicking the Rows | Columns selector.

You can specify any of the following bucket aggregations for your pie chart:

Date Histogram

A date histogram is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second.

Histogram

A standard histogram is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a range aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A date range aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using date math expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The IPv4 range aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A terms aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of filters for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the labelbutton label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental significant terms aggregation. The value of the Size parameter defines the number of entries this aggregation returns.

After defining an initial bucket aggregation, you can define sub-aggregations to refine the visualization. Click + Add Sub Aggregation to define a sub-aggregation, then choose Split Slices to select a sub-aggregation from the list of types.

When multiple aggregations are defined on a chart’s axis, you can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

You can customize the colors of your visualization by clicking the color dot next to each label to display the color picker.

An array of color dots that users can select

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Exclude Pattern Flags

A standard set of Java flags for the exclusion pattern.

Include Pattern

Specify a pattern in this field to include in the results.

Include Pattern Flags

A standard set of Java flags for the inclusion pattern.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the table:

Donut

Display the chart as a sliced ring instead of a sliced pie.

Show Tooltip

Check this box to enable the display of tooltips.

Show Legend

Check this box to enable the display of a legend next to the chart.

After changing options, click the green Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Tile Maps

A tile map displays a geographic area overlaid with circles keyed to the data determined by the buckets you specify.

Note
By default, Kibi uses the Elastic Tile Service to display map tiles. To use other tile service providers, configure the tilemap settings in kibi.yml.

The default metrics aggregation for a tile map is the Count aggregation. You can select any of the following aggregations as the metrics aggregation:

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, specify if you are splitting the chart or displaying the buckets as Geo Coordinates on a single chart. A multiple chart split must run before any other aggregations.

Tile maps use the Geohash aggregation as their initial aggregation. Select a field, typically coordinates, from the drop-down. The Precision slider determines the granularity of the results displayed on the map. See the documentation for the geohash grid aggregation for details on the area specified by each precision level. Kibana supports a maximum geohash length of 7.

Note
Higher precisions increase memory usage for the browser displaying Kibi as well as for the underlying Elasticsearch cluster.

Once you’ve specified a buckets aggregation, you can define sub-aggregations to refine the visualization. Tile maps only support sub-aggregations as split charts. Click + Add Sub Aggregation, then Split Chart to select a sub-aggregation from the list of types:

Date Histogram

A date histogram is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second.

Histogram

A standard histogram is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a range aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range. After changing options, click the green Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Date Range

A date range aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using date math expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The IPv4 range aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A terms aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of filters for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the labelbutton label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental significant terms aggregation. The value of the Size parameter defines the number of entries this aggregation returns.

Geohash

The geohash aggregation displays points based on the geohash coordinates.

Note
By default, the Change precision on map zoom box is checked. Uncheck the box to disable this behavior.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Exclude Pattern Flags

A standard set of Java flags for the exclusion pattern.

Include Pattern

Specify a pattern in this field to include in the results.

Include Pattern Flags

A standard set of Java flags for the inclusion pattern.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the chart:

Map type

Select one of the following options from the drop-down.

Scaled Circle Markers

Scale the size of the markers based on the metric aggregation’s value.

Shaded Circle Markers

Displays the markers with different shades based on the metric aggregation’s value.

Shaded Geohash Grid

Displays the rectangular cells of the geohash grid instead of circular markers, with different shades based on the metric aggregation’s value.

Heatmap

A heat map applies blurring to the circle markers and applies shading based on the amount of overlap. Heatmaps have the following options:

  • Radius: Sets the size of the individual heatmap dots.

  • Blur: Sets the amount of blurring for the heatmap dots.

  • Maximum zoom: Tilemaps in Kibi support 18 zoom levels. This slider defines the maximum zoom level at which the heatmap dots appear at full intensity.

  • Minimum opacity: Sets the opacity cutoff for the dots.

  • Show Tooltip: Check this box to have a tooltip with the values for a given dot when the cursor is on that dot.

Desaturate map tiles

Desaturate the map’s color in order to make the markers stand out more clearly.

WMS compliant map server

Check this box to enable the use of a third-party mapping service that complies with the Web Map Service (WMS) standard. Specify the following elements:

  • WMS url: The URL for the WMS map service.

  • WMS layers: A comma-separated list of the layers to use in this visualization. Each map server provides its own list of layers.

  • WMS version: The WMS version used by this map service.

  • WMS format: The image format used by this map service. The two most common formats are image/png and image/jpeg.

  • WMS attribution: An optional, user-defined string that identifies the map source. Maps display the attribution string in the lower right corner.

  • WMS styles: A comma-separated list of the styles to use in this visualization. Each map server provides its own styling options.

After changing options, click the green Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Once your tilemap visualization is ready, you can explore the map in several ways:

  • Click and hold anywhere on the map and move the cursor to move the map center. Hold Shift and drag a bounding box across the map to zoom in on the selection.

  • Click the Zoom In/Out viz-zoom buttons to change the zoom level manually.

  • Click the Fit Data Bounds viz-fit-bounds button to automatically crop the map boundaries to the geohash buckets that have at least one result.

  • Click the Latitude/Longitude Filter viz-lat-long-filter button, then drag a bounding box across the map, to create a filter for the box coordinates.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Vertical Bar Charts

This chart’s Y axis is the metrics axis. The following aggregations are available for this axis:

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Percentiles

The percentile aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The percentile ranks aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

You can add an aggregation by clicking the + Add Aggregation button.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, specify if you are splitting slices within a single chart or splitting into multiple charts. A multiple chart split must run before any other aggregations. When you split a chart, you can change if the splits are displayed in a row or a column by clicking the Rows | Columns selector.

The X axis of this chart is the buckets axis. You can define buckets for the X axis, for a split area on the chart, or for split charts.

This chart’s X axis supports the following aggregations. Click the linked name of each aggregation to visit the main Elasticsearch documentation for that aggregation.

Date Histogram

A date histogram is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second.

Histogram

A standard histogram is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a range aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A date range aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using date math expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The IPv4 range aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A terms aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of filters for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the Label button icon label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental significant terms aggregation.

External query terms filter

a Kibi aggregator where one can define one or more buckets based on some record value (typically a primary key) matching the results of an external query. Multiple such buckets, corresponding to multiple queries, can be defined. For more information see the query menu in the configuration. This displays the results of the external query terms filter aggregation.

Once you’ve specified an X axis aggregation, you can define sub-aggregations to refine the visualization. Click + Add Sub Aggregation to define a sub-aggregation, then choose Split Area or Split Chart, then select a sub-aggregation from the list of types.

When multiple aggregations are defined on a chart’s axis, you can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

Enter a string in the Custom Label field to change the display label.

You can customize the colors of your visualization by clicking the color dot next to each label to display the color picker.

An array of color dots that users can select

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Exclude Pattern Flags

A standard set of Java flags for the exclusion pattern.

Include Pattern

Specify a pattern in this field to include in the results.

Include Pattern Flags

A standard set of Java flags for the inclusion pattern.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

The availability of these options varies depending on the aggregation you choose.

Select the Options to change the following aspects of the table:

Bar Mode

When you have multiple Y-axis aggregations defined for your chart, you can use this drop-down to affect how the aggregations display on the chart:

stacked

Stacks the aggregations on top of each other.

percentage

Displays each aggregation as a proportion of the total.

grouped

Groups the results horizontally by the lowest-priority sub-aggregation.

Checkboxes are available to enable and disable the following behaviors:

Show Tooltip

Check this box to enable the display of tooltips.

Show Legend

Check this box to enable the display of a legend next to the chart.

Scale Y-Axis to Data Bounds

The default Y axis bounds are zero and the maximum value returned in the data. Check this box to change both upper and lower bounds to match the values returned in the data.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana field formatters.

Kibi Word Cloud

The Kibi Wordcloud visualization displays the most frequent terms in the current set of Elasticsearch documents at a glance.

To configure the visualization, select Count as the aggregation for the metric, then add a Split Rows bucket configuration; you can choose to process terms using either the Terms aggregation or the Significant Terms aggregation.

Using the terms aggregation

If the bucket aggregation is set to Terms, you need to specify the following parameters:

  • Field: the field which will provide terms to be aggregated.

  • Order and Size: the subset and number of terms to be aggregated.

  • Order By: the metric used to sort the terms.

For example, to display the Top 30 terms by count in the snippet field of an index containing articles, set the following configuration:

Terms aggregation configuration

To exclude common words or any unwanted term, click on Advanced and and write a regular expression in the Exclude pattern field:

Terms aggregation advanced configuration

Using the significant terms aggregation

If the bucket aggregation is set to Significant Terms, you need to specify the following parameters:

  • Field: the field which will provide terms to be aggregated.

  • Size: the number of significant terms to be aggregated.

Significant Terms aggregation advanced configuration

Kibi Timeline

The Kibi Timeline visualization displays series of data coming from different saved searches on a single timeline component. Events are color coded to distinguish between different groups.

Each event on a timeline become a clickable term filter which allow to quickly filter the related data based on what is shown on the timeline.

Timeline

Configuration

To configure the visualization, add a new Group and select:

  • Saved search id - date for this group will be taken from corresponding index.

  • Group label - a label for the group.

  • Event label field - field value will be used as individual event label.

  • Event start date - date from this field will be used to position start of the event.

  • Event end date - (optional) date from this field will be used to position end of the event.

  • Events number limit - (optional) limit number of events in this group.

Timeline configuration

Advanced option

By default events from multiple groups are rendered all mixed together. It is possible to show different groups on different levels by enabling the advanced option

  • Groups rendered on separate levels

Timeline advanced configuration

Below timeline where each group is rendered on separate level

Timeline

Radar Chart

A radar chart introduced in kibi-0.3.0 is a graphical method of displaying multivariate data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point. The relative position and angle of the axes is typically uninformative.

Radar chart visualization
Radar chart settings

The radar chart is also known as web chart, spider chart, star chart. It is developed as a standalone plugin suitable to install in both Kibi 0.3+ and Kibana 4.3+.

Kibi Graph Browser [Enterprise Edition only]

NOTE: Documentation for Kibi Graph Browser is available only in Kibi Enterprise Edition.

Kibi Box Plot [Enterprise Edition only]

NOTE: Documentation for Kibi Box Plot is available only in Kibi Enterprise Edition.

Bubble Diagram [Enterprise Edition only]

NOTE: Documentation for Kibi Bubble Diagram visualisation is available only in Kibi Enterprise Edition.

Kibi Scatter Plot [Enterprise Edition only]

NOTE: Documentation for Kibi Scatter Plot is available only in Kibi Enterprise Edition.

Kibi Vector Map [Enterprise Edition only]

NOTE: Documentation for Kibi Vector Map is available only in Kibi Enterprise Edition.

Kibi Horizontal Bar Chart [Enterprise Edition only]

NOTE: Documentation for Kibi Horizontal Bar Chart is available only in Kibi Enterprise Edition.

Kibi Multi Chart [Enterprise Edition only]

NOTE: Documentation for Kibi Multi Chart is available only in Kibi Enterprise Edition.

Kibi Relational Browsing

Kibi adds a relational dimension to Kibana when browsing indices, allowing to filter documents on a dashboard by showing only those that have a relation with documents displayed in a different dashboard, possibly stored in different indices.

There are two paradigms a user can follow:

  1. through the Relational Panel, a user picks a set of dashboards to join: filtering documents within a dashboard automatically updates the other related dashboards; and

  2. through the Relational filter, a user builds interactivelly the sequence of dashboards to join.

Relational Panel

With the relational panel browsing paradigm, all the dashboards of interest are join together. Effectivelly, documents that can be viewed in each dashboard are those that are connected with the rest.

For example, consider two dashboards Articles and Companies which documents are related to one another; if one filters articles by their publishing date to a certain range, then only the companies that are mentioned in articles published within that range are displayed.

Configuration

In order to use this browsing paradigm, it is necessary to configure the relationships.

Once this is done, you can view which dashboards are connected to each other using the relational panel by clicking on the icon Relational panel icon in the Dashboard tab. In the image below, the dashboards Articles, Companies, and Investments are connected together.

Relational panel

Browsing

Since a saved search is now associated with a dashboard, a count indicating the current number of documents in that dashboard is displayed on each tab. For example on the previous image, there are 151714 articles, 6419 companies, and 13231 investments rounds that are inter-connected. These numbers coupled with the relational panel allows to get a feeling of the current state of the joined dashboards. The current set of connected dashboards are displayed in the label of the blue filter relational panel filter.

Since all the dashboards in the set are connected to each other, adding a filter to a dashboard will also update the connected dashboards. If you select TechCrunch as the source for articles, there are now in total 27772 articles. Being automatically updated, we observe that only 3748 companies (out of 6419) and 8136 investments (out of 13231) are actually connected to those articles from TechCrunch.

Relational panel with a filter

You can see the current filters that are applied with the relational panel by hovering the mouse over the blue filter. The explanation tooltip shown below indicates that the current 311 companies are connected with:

  • articles coming from TechCrunch published within a specific time range; and

  • investments funded within another time range.

Explanation of the relational filter

Relational filter

The relational filter visualization allows to "pivot" from a dashboard to another by creating a join between multiple indices based on their relations. This allows to interactivelly build the sequence of dashboards to join.

The relational filter visualzation is configured based on the relationships between indices defined in the settings tab. For example, let’s take the following indices:

article

an index containing articles; each document in the index has a field called companies which is an array that contains the ID of companies mentioned in the article. This index is displayed on the dashboard entitled Articles.

company

an index containing information about companies; each document in the index has a field called id that contains the ID of the company. This index is displayed on the dashboard entitled Companies.

Both indices are configured so that they are joined on the field companies of article with the the field id of company. Then, it is possible to use that configuration in order to create a relational filter that would filter companies based on connected articles (or vice-versa).

In the Articles dashboard, the relational filter visualization is displayed as a button which indicates the number of documents in the Companies dashboard that are mentioned in the articles of the current dashboard.

The screenshot below shows the button for the relation described in the example; there are 18237 companies mentioned in the 618155 articles currently displayed:

Relational filter button on the Articles dashboard

Clicking on the button will switch you to the Companies dashboard and display the 18237 companies; the relational filter is displayed in the filter bar, as displayed below:

Relational filter on the Companies dashboard

The relational filter visualization requires the Siren Join plugin 2.4.4 for Elasticsearch. The plugin is compatible with Elasticsearch 2.4.4.

For more information about the Siren Join plugin visit our website at https://siren.solutions/searchplugins/join/.

Configuration

Click on the Add filter button to create a new filter in the visualization; the filter is defined by the following parameters:

  • Button label: the label of the button that will be displayed inside the visualization, e.g. Companies -→.

  • Custom filter label: the label of the filter that will be displayed in the filter bar, which by default is …​ related to ($COUNT) from $DASHBOARD.. Several variables are available for customizing the label:

    • $COUNT is a number of items on source dashboard,

    • $DASHBOARD is a source dashboard name.

  • Source dashboard: optional parameter that indicates on which dashboard the relational filter should appear in.

  • Target dashboard: the dashboard to join the current dashboard with. The current dashboard is equal to the previous field if set.

  • Relation: the label of the relation between indices to use for this relational filter. This is set in the relations settings tab.

The screenshot below shows the configuration of a relation from the Articles dashboard to the Companies dashboard, using the mentions relation:

Relational filter configuration

It is possible to define multiple relations in a single Kibi relational filter visualization; the visualization will display only buttons applicable to the currently displayed dashboard.

Usage

When clicking on a button in the relational filter visualization, the current state of the source dashboard is added to the relational filter and applied to the target dashboard. Just move the mouse over relational filter to see an explanation of what is being joined.

Walkthrough example

We start on the Articles dashboard, search for pizza and click on the relational filter to switch to the Companies dashboard.

Relational filter explanation

Hovering over the blue filter displays an explanation. It indicates that the relational filter involves only one join, i.e., the one from Articles to Companies with pizza filtering the articles.

Relational filter explanation

Next, we add a regular filter to the Companies dashboard by clicking on the USA row of the Companies by Country visualization.

Relational filter explanation

Now, we click on the Investment rounds -→ button which takes us to the Investment rounds dashboard. The explanation on that filter shows that the investment rounds are filtered as follows:

  • the current investments rounds are joined with companies from the USA; and

  • those companies are joined with articles which match the term pizza.

Relational filter explanation
Note
The sequence of the joins in the explanation are shown in reverse, i.e., the last join is on top.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization, as in this example:

Spy panel of the relational filter visualization

This panel provides two kinds data: information about the query behind the relational filter in the Multi Search tab, and details about the visualization object in the Debug tab.

This pane presents information about the msearch request executed to perform the joins. A relational filter corresponds to one query of the msearch.

On the top, the time reported in Multi search request duration informs on how long the msearch request took. There is also additional information about each query of the msearch:

  • Query Duration: The time spent for this particular query.

  • Hits: the total number of documents resulting from the query.

  • Index: the index pattern used to execute the query.

  • Type: the type of the indices matched by the index pattern.

For a particular relational filter, you can get additional information about the query that got executed.

Filterjoin

This displays a table that provides several statistics about each join.

Details about the filterjoin query
Raw Request

The filterjoin query as sent by Kibi. This uses the internal API for defining the join.

Translated Request

The filterjoin query as sent to the Elasticsearch cluster, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Debug

The Debug tab presents the JSON object that Kibi uses for this relational filter.

Debug spy panel of the relational filter visualization

Dashboard

A Kibi dashboard displays a set of saved visualizations in a customizable grid layout. You can save a dashboard to share or reload at a later time.

In Kibi, dashboards are displayed as tabs and can be organized as dashboard groups.

Getting Started

You need at least one saved visualization to use a dashboard.

Building a New Dashboard

The first time you click the Dashboard tab, Kibi displays the first available dashboard or, if no dashboards have been defined, the dashboard creation screen.

New Dashboard screen

Build your dashboard by adding visualizations. By default, Kibana dashboards use a light color theme. To use a dark color theme instead, click the Settings Gear button and check the Use dark theme box.

Dark Theme Example

Note
You can change the default theme in the Advanced section of the Settings tab.

Saving Dashboards

To save the dashboard, click the Save Dashboard Save button:

Saving a dashboard

The name of the dashboard can be set in the Save As field.

If Store time with dashboard is checked, the time filter currently set will be restored when the dashboard is opened.

To display the number of Elasticsearch documents displayed by the dashboard in the corresponding tab, select a Saved Search:

Dashboard settings

Sharing Dashboards

You can share dashboards with other users by sending a link or by embedding them into HTML pages; make sure that your Kibi installation is properly secured when sharing a dashboard on a public facing server.

Note
To view shared dashboards users must be able to access Kibi; keep this in mind if your Kibi instance is protected by an authentication proxy.

To share a dashboard, click the Share button share-dashboard to display the Sharing panel.

sharing-panel

Click the Copy to Clipboard button share-link to copy the native URL or embed HTML to the clipboard. Click the Generate short URL button share-short-link to create a shortened URL for sharing or embedding.

Embedding Dashboards

To embed a dashboard, copy the embed code from the Share display into your external web application.

Adding Visualizations to a Dashboard

To add a visualization to the dashboard, click the Add Visualization Plus button in the toolbar panel, then select a previously created visualization from the list:

Adding a visualization to the dashboard

You can filter the list of visualizations by typing a filter string into the Visualization Filter field.

The visualization you select appears in a container on your dashboard.

Note
If you see a message about the container’s height or width being too small, resize the container.

Reset all dashboards to their default state

One can save with dashboard some specific filters, a custom query or a certain time range. If you click on the Reset the time, filters, and queries from all dashboards to their default saved state Minus button in the toolbar panel, the temporary filters/queries/time set on all dashboards would be removed, reverted to a dashboard’s default state with the saved filters/query/time.

Note
If the relational panel is enabled, clicking this button does not remove the filter. If you want to do so, you need to disable the relational panel in the Relations settings.

Customizing Dashboard Elements

The visualizations in your dashboard are stored in resizable containers that you can arrange on the dashboard. This section discusses customizing these containers.

Moving Containers

Click and hold a container’s header to move the container around the dashboard. Other containers will shift as needed to make room for the moving container. Release the mouse button to confirm the container’s new location.

Resizing Containers

Move the cursor to the bottom right corner of the container until the cursor changes to point at the corner. After the cursor changes, click and drag the corner of the container to change the container’s size. Release the mouse button to confirm the new container size.

Removing Containers

Click the x icon at the top right corner of a container to remove that container from the dashboard. Removing a container from a dashboard does not delete the saved visualization in that container.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization, as in this example:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

NYCTA-Table

Request

The raw request used to query the server, presented in JSON format.

NYCTA-Request

Response

The raw response from the server, presented in JSON format.

NYCTA-Response

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

NYCTA-Statistics

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Kibana [field formatters].

Changing the Visualization

Click the Edit button Pencil button at the top right of a container to open the visualization in the Visualize page.

Working with Filters

When you create a filter anywhere in Kibi, the filter conditions display in a green oval under the search text entry box:

filter sample

Hovering on the filter oval displays the following icons:

filter allbuttons
Enable Filter filter-enable

Click this icon to disable the filter without removing it. You can enable the filter again later by clicking the icon again. Disabled filters display a striped shaded color, green for inclusion filters and red for exclusion filters.

Pin Filter filter-pin

Click this icon to pin a filter. Pinned filters persist across Kibi tabs. You can pin filters from the Visualize tab, click on the Discover or Dashboard tabs, and those filters remain in place.

Note
If you have a pinned filter and you’re not seeing any query results, check that your current tab’s index pattern is one that the filter applies to. E.g. a filter "name:giovanni" will results in 0 results if pinned and therefore "dragged along" to a dashboard whose underlying index does not have a "name" field, let alone a "giovanni" value. For this reason a good pattern in Kibi is to use Dashboard Groups to group together dashboard which are based on the same underlying index. In this case the user can safely pin and "drag along" a filter across dashboards in the same group.
Toggle Filter filter-toggle

Click this icon to toggle a filter. By default, filters are inclusion filters, and display in green. Only elements that match the filter are displayed. To change this to an exclusion filters, displaying only elements that don’t match, toggle the filter. Exclusion filters display in red.

Remove Filter filter-delete

Click this icon to remove a filter entirely.

Custom Filter filter-custom

Click this icon to display a text field where you can customize the JSON representation of the filter and specify an alias to use for the filter name:

filter custom json

You can use JSON filter representation to implement predicate logic, with should for OR, must for AND, and must_not for NOT:

Example 7. OR Example
{
  "bool": {
    "should": [
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "China"
        }
      }
    ]
  }
}
Example 8. AND Example
{
  "bool": {
    "must": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.city_name.raw": "New York"
        }
      }
    ]
  }
}
Example 9. NOT Example
{
  "bool": {
    "must_not": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      }
    ]
  }
}

Click the Done button to update the filter with your changes.

To apply any of the filter actions to all the filters currently in place, click the filter-actions Global Filter Actions button and select an action.

Dashboard Groups

Dashboards can be organized in dashboard groups; dashboard groups are displayed as tabs just like dashboards.

Clicking on a dashboard group will display the first dashboard in the group by default; the other dashboards in the group can be selected through the dropdown menu embedded in the tab:

A dashboard group tab

If the dashboard is associated with a saved search the count of documents on the dashboard is displayed next to the dashboard name. Two additional indicators that might be displayed are:

  • Filters/Queries indicator - the filter icon is displayed if there are any filter or query on the dashboard

  • Pruned joins indicator - a star symbol is displayed if any of the join operations was pruned

An example image below was taken on the Articles detailed per source dashboard. By opening the select dropdown, we see that there are 7225 documents on Articles dashboard. We also know that there are some filters or queries on this dashboard and at least one of the join operations was pruned.

A dashboard counts and indicators

Dashboard groups can be managed by clicking on the Dashboard Groups Settings tab.

Automatically Refreshing the Page

You can configure a refresh interval to automatically refresh the page with the latest index data. This periodically resubmits the search query.

When a refresh interval is set, it is displayed to the left of the Time Filter in the menu bar.

To set the refresh interval:

  1. Click the Time Filter Time Filter in the upper right corner of the menu bar.

  2. Click the Refresh Interval tab.

  3. Choose a refresh interval from the list.

To automatically refresh the data, click the autorefresh Auto-refresh button and select an autorefresh interval:

autorefresh intervals

When auto-refresh is enabled, Kibi’s top bar displays a pause button and the auto-refresh interval: autorefresh-pause. Click the Pause button to pause auto-refresh.

Settings

Indices

To use Kibi, you have to tell it about the Elasticsearch indices that you want to explore by configuring one or more index patterns. You can also:

  • Create scripted fields that are computed on the fly from your data. You can browse and visualize scripted fields, but you cannot search them.

  • Set advanced options such as the number of rows to show in a table and how many of the most popular fields to show. Use caution when modifying advanced options, as it’s possible to set values that are incompatible with one another.

  • Configure Kibi for a production environment

Creating an Index Pattern to Connect to Elasticsearch

An index pattern identifies one or more Elasticsearch indices that you want to explore with Kibi. Kibi looks for index names that match the specified pattern. An asterisk () in the pattern matches zero or more characters. For example, the pattern myindex- matches all indices whose names start with myindex-, such as myindex-1 and myindex-2.

An index pattern can also simply be the name of a single index.

To create an index pattern to connect to Elasticsearch:

  1. Go to the Settings > Indices tab.

  2. Specify an index pattern that matches the name of one or more of your Elasticsearch indices. By default, Kibi guesses that you’re you’re working with log data being fed into Elasticsearch by Logstash.

    Note
    When you switch between top-level tabs, Kibi remembers where you were. For example, if you view a particular index pattern from the Settings tab, switch to the Discover tab, and then go back to the Settings tab, Kibi displays the index pattern you last looked at. To get to the create pattern form, click the Add button in the Index Patterns list.
  3. If your index contains a timestamp field that you want to use to perform time-based comparisons, select the Index contains time-based events option and select the index field that contains the timestamp. Kibi reads the index mapping to list all of the fields that contain a timestamp.

  4. By default, Kibi restricts wildcard expansion of time-based index patterns to indices with data within the currently selected time range. Click Do not expand index pattern when search to disable this behavior.

  5. Click Create to add the index pattern.

  6. To designate the new pattern as the default pattern to load when you view the Discover tab, click the favorite button.

Note
When you define an index pattern, indices that match that pattern must exist in Elasticsearch. Those indices must contain data.

To use an event time in an index name, enclose the static text in the pattern and specify the date format using the tokens described in the following table.

For example, [logstash-]YYYY.MM.DD matches all indices whose names have a timestamp of the form YYYY.MM.DD appended to the prefix logstash-, such as logstash-2015.01.31 and logstash-2015-02-01.

Date Format Tokens
M

Month - cardinal: 1 2 3 …​ 12

Mo

Month - ordinal: 1st 2nd 3rd …​ 12th

MM

Month - two digit: 01 02 03 …​ 12

MMM

Month - abbreviation: Jan Feb Mar …​ Dec

MMMM

Month - full: January February March …​ December

Q

Quarter: 1 2 3 4

D

Day of Month - cardinal: 1 2 3 …​ 31

Do

Day of Month - ordinal: 1st 2nd 3rd …​ 31st

DD

Day of Month - two digit: 01 02 03 …​ 31

DDD

Day of Year - cardinal: 1 2 3 …​ 365

DDDo

Day of Year - ordinal: 1st 2nd 3rd …​ 365th

DDDD

Day of Year - three digit: 001 002 …​ 364 365

d

Day of Week - cardinal: 0 1 3 …​ 6

do

Day of Week - ordinal: 0th 1st 2nd …​ 6th

dd

Day of Week - 2-letter abbreviation: Su Mo Tu …​ Sa

ddd

Day of Week - 3-letter abbreviation: Sun Mon Tue …​ Sat

dddd

Day of Week - full: Sunday Monday Tuesday …​ Saturday

e

Day of Week (locale): 0 1 2 …​ 6

E

Day of Week (ISO): 1 2 3 …​ 7

w

Week of Year - cardinal (locale): 1 2 3 …​ 53

wo

Week of Year - ordinal (locale): 1st 2nd 3rd …​ 53rd

ww

Week of Year - 2-digit (locale): 01 02 03 …​ 53

W

Week of Year - cardinal (ISO): 1 2 3 …​ 53

Wo

Week of Year - ordinal (ISO): 1st 2nd 3rd …​ 53rd

WW

Week of Year - two-digit (ISO): 01 02 03 …​ 53

YY

Year - two digit: 70 71 72 …​ 30

YYYY

Year - four digit: 1970 1971 1972 …​ 2030

gg

Week Year - two digit (locale): 70 71 72 …​ 30

gggg

Week Year - four digit (locale): 1970 1971 1972 …​ 2030

GG

Week Year - two digit (ISO): 70 71 72 …​ 30

GGGG

Week Year - four digit (ISO): 1970 1971 1972 …​ 2030

A

AM/PM: AM PM

a

am/pm: am pm

H

Hour: 0 1 2 …​ 23

HH

Hour - two digit: 00 01 02 …​ 23

h

Hour - 12-hour clock: 1 2 3 …​ 12

hh

Hour - 12-hour clock, 2 digit: 01 02 03 …​ 12

m

Minute: 0 1 2 …​ 59

mm

Minute - two-digit: 00 01 02 …​ 59

s

Second: 0 1 2 …​ 59

ss

Second - two-digit: 00 01 02 …​ 59

S

Fractional Second - 10ths: 0 1 2 …​ 9

SS

Fractional Second - 100ths: 0 1 …​ 98 99

SSS

Fractional Seconds - 1000ths: 0 1 …​ 998 999

Z

Timezone - zero UTC offset (hh:mm format): -07:00 -06:00 -05:00 .. +07:00

ZZ

Timezone - zero UTC offset (hhmm format): -0700 -0600 -0500 …​ +0700

X

Unix Timestamp: 1360013296

x

Unix Millisecond Timestamp: 1360013296123

Setting the Default Index Pattern

The default index pattern is loaded by automatically when you view the Discover tab. Kibi displays a star to the left of the name of the default pattern in the Index Patterns list on the Settings > Indices tab. The first pattern you create is automatically designated as the default pattern.

To set a different pattern as the default index pattern:

  1. Go to the Settings > Indices tab.

  2. Select the pattern you want to set as the default in the Index Patterns list.

  3. Click the pattern’s Favorite button.

Note
You can also manually set the default index pattern in advanced settings.

Reloading the Index Fields List

When you add an index mapping, Kibi automatically scans the indices that match the pattern to display a list of the index fields. You can reload the index fields list to pick up any newly-added fields.

Reloading the index fields list also resets Kibi’s popularity counters for the fields. The popularity counters keep track of the fields you’ve used most often within Kibi and are used to sort fields within lists.

To reload the index fields list:

  1. Go to the Settings > Indices tab.

  2. Select an index pattern from the Index Patterns list.

  3. Click the pattern’s Reload button.

Deleting an Index Pattern

To delete an index pattern:

  1. Go to the Settings > Indices tab.

  2. Select the pattern you want to remove in the Index Patterns list.

  3. Click the pattern’s Delete button.

  4. Confirm that you want to remove the index pattern.

Managing Fields

The fields for the index pattern are listed in a table. Click a column header to sort the table by that column. Click the Controls button in the rightmost column for a given field to edit the field’s properties. You can manually set the field’s format from the Format drop-down. Format options vary based on the field’s type.

You can also set the field’s popularity value in the Popularity text entry box to any desired value. Click the Update Field button to confirm your changes or Cancel to return to the list of fields.

Kibi has field formatters for the following field types:

String Field Formatters

String fields support the String and Url formatters.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

The Url field formatter can take on the following types:

  • The Link type turn the contents of the field into an URL.

  • The Image type can be used to specify an image directory where a specified image is located.

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

In order to pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Date Field Formatters

Date fields support the Date, Url, and String formatters.

The Date formatter enables you to choose the display format of date stamps using the moment.js standard format definitions.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

The Url field formatter can take on the following types:

  • The Link type turn the contents of the field into an URL.

  • The Image type can be used to specify an image directory where a specified image is located.

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

In order to pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Geographic Point Field Formatters

Geographic point fields support the String formatter.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

Numeric Field Formatters

Numeric fields support the Url, String, Bytes, Number, Percentage, and Color formatters.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

The Url field formatter can take on the following types:

  • The Link type turn the contents of the field into an URL.

  • The Image type can be used to specify an image directory where a specified image is located.

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

In order to pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

The Color field formatter enables you to specify colors with specific ranges of values for a numeric field.

When you select the Color field formatter, Kibana displays the Range, Font Color, Background Color, and Example fields.

Click the Add Color button to add a range of values to associate with a particular color. You can click in the Font Color and Background Color fields to display a color picker. You can also enter a specific hex code value in the field. The effect of your current color choices are displayed in the Example field.

colorformatter

The Bytes, Number, and Percentage formatters enable you to choose the display formats of numbers in this field using the numeral.js standard format definitions.

Creating a Scripted Field

Scripted fields compute data on the fly from the data in your Elasticsearch indices. Scripted field data is shown on the Discover tab as part of the document data, and you can use scripted fields in your visualizations. Scripted field values are computed at query time so they aren’t indexed and cannot be searched.

Note
Kibana cannot query scripted fields.
Warning
Computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Kibi’s performance. Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you’ll get exceptions whenever you try to view the dynamically generated data.

Scripted fields use the Lucene expression syntax. For more information, see Lucene Expressions Scripts.

You can reference any single value numeric field in your expressions, for example:

doc['field_name'].value

To create a scripted field:

  1. Go to Settings > Indices

  2. Select the index pattern you want to add a scripted field to.

  3. Go to the pattern’s Scripted Fields tab.

  4. Click Add Scripted Field.

  5. Enter a name for the scripted field.

  6. Enter the expression that you want to use to compute a value on the fly from your index data.

  7. Click Save Scripted Field.

For more information about scripted fields in Elasticsearch, see Scripting.

Note
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable dynamic Groovy scripting.

Updating a Scripted Field

To modify a scripted field:

  1. Go to Settings > Indices

  2. Click the Edit button for the scripted field you want to change.

  3. Make your changes and then click Save Scripted Field to update the field.

Warning
Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you’ll get exceptions whenever you try to view the dynamically generated data.

Deleting a Scripted Field

To delete a scripted field:

  1. Go to Settings > Indices

  2. Click the Delete button for the scripted field you want to remove.

  3. Confirm that you really want to delete the field.

Source Filtering

Some index fields are useful for filtering or aggregation purposes, but are not used in visualizations like the Enhanced search results.

In addition, they can negatively impact on the performance of the application if they contain large amounts of data, creating additional network traffic.

The Source filtering API of Elasticsearch allows to control which fields are included in the response.

Source filtering can be configured in the index pattern page by clicking on the Retrieved fields tab.

Index view

Any visualisation based on the index pattern will receive a filtered response.

Filtering rules can be configured by writing a JSON object to the configuration form.

The JSON object can contain two attributes, exclude and include, as explained in the Source filtering API.

The screenshot below displays a filtering configuration that excludes the snippet field from the _source.

Source filtering configuration

The column retrieved in the Fields tab specifies if the values associated with a field should be retrieved or not according to the filtering rules.

Relations

In this panel, you can define relationships between index patterns. These relationships ultimately form a graph of index patterns. This graph is used in conjunction with the Siren 2.0 plugin; this allows to perform join operations between dashboards, i.e., filtering a dashboard’s documents with regards to an other.

Graph of Index Patterns

A relationship is defined as a join operation between two indices with the following fields:

  • Left Index Pattern: the left index of the join;

  • Left Type: the type of the left index of the join;

  • Left Field: the field of the left index to join on;

  • Right Index Pattern: the right index of the join;

  • Right Type: the type of the right index of the join;

  • Right Field: the field of the right index to join with; and

  • Label: the label of the relation.

The image below displays a graph of four index patterns, where three relationships have defined. You can add a new relationship by clicking on the "Add relation" button.

Graph of Index Patterns

A relation is also indicated on the Indices tab thanks to the icon Index relation icon. If you hover the mouse over it, a tooltip message is displayed indicating the index pattern and field that field is joined with.

For example, in the image below, that icon is displayed next to the field "id" of the "investor" index, which is join with the field "investorid" of the "investment" index.

Investor Index
Advanced settings for index relations [Enterprise Edition only]
NOTE: Documentation for Advanced settings for index relations is available only in Kibi Enterprise Edition.

Graph of Dashboards

Based on the graph of index patterns, one can define relationships between dashboards, since a dashboard may be associated with a saved search.

A relationship between dashboards is defined with the following parameters

  • Left Dashboard and Right Dashboard: the dashboards to be linked; and

  • Relation: the relation that connects the two dashboards; it is the label of a relation previously defined.

For example, the image below shows a graph of five dashboards:

  1. the dashboard "Articles" with a saved search associated to the index "article" depicted with the circle ;

  2. the dashboard "Articles detailed per source" with a saved search associated to the index "article" depicted with the circle ;

  3. the dashboard "Companies" with a saved search associated to the index "company" depicted with the circle ;

  4. the dashboard "Investments" with a saved search associated to the index "investment" depicted with the circle ; and

  5. the dashboard "Investors" with a saved search associated to the index "investor" depicted with the circle .

Such relationships between dashboards are defined on the right side of the panel, as shown in the image below.

Graph of Dashboards
Enabling Relations

A relation between two dashboard can be enabled or disabled. If it is enabled, the documents are filtered based on that relation. You can disable/enable a relation by checking the box on an edge of the graph.

In the previous image, the AC and CI relations between the dashboards Articles, Companies, and Investments are enabled. This means that filtering documents from the Articles will retain only the Companies, and Investments, that are connected to those articles.

Enabling Relational Panel

By checking the "Enable relational panel" checkbox, the relationships that you defined between dashboards will be made effective while browsing. The relational panel can be opened by clicking on the icon Relational panel icon in the Dashboard tab.

With relations between dashboards enabled in the previous example, you will see the "relational" filter Relational filter displayed on the dashboards Articles, Companies, and Investments.

Datasources

For an overview of datasources, please read the External datasources chapter.

Queries

For an overview of queries, please read the External datasources chapter.

Query Templates

For an overview of query templates, please read the External datasources chapter.

Dashboard Groups

Dashboard groups can be managed through the Dashboard Groups tab.

You can create a new group by clicking on the New Dashboard Group New button, save it with the Save Dashboard Group Save button, or load an existing one with the Load Dashboard Group Load button.

You can set the following parameters for a dashboard group:

  • Title: the title of the group, displayed on the tab.

  • Description: an optional description of the group.

  • Priority: a number that determines the position of the tab.

  • Icon URL/Icon CSS class: allows to set the icon displayed in the tab either as a PNG image or a CSS font class; Kibi includes the Font Awesome toolkit.

Advanced

The Advanced Settings page enables you to directly edit settings that control the behavior of the Kibi application. For example, you can change the format used to display dates, specify the default index pattern, and set the precision for displayed decimal values.

Warning
Changing advanced settings can have unintended consequences. If you aren’t sure what you’re doing, it’s best to leave these settings as-is.

To set advanced settings:

  1. Go to Settings > Advanced.

  2. Click the Edit button for the option you want to modify.

  3. Enter a new value for the option.

  4. Click the Save button.

Query string options

You can set the default options used when performing a query_string query thanks to the advanced option query:queryString:options.

Warning
Modifying the following settings can significantly affect Kibana’s performance and cause problems that are difficult to diagnose. Setting a property’s value to a blank field will revert to the default behavior, which may not be compatible with other configuration settings. Deleting a custom setting removes it from Kibana permanently.
Kibana Settings Reference
query:queryString:options

Options for the Lucene query string parser.

sort:options

Options for the Elasticsearch sort parameter.

dateFormat

The format to use for displaying pretty-formatted dates.

dateFormat:tz

The timezone that Kibana uses. The default value of Browser uses the timezone detected by the browser.

dateFormat:scaled

These values define the format used to render ordered time-based data. Formatted timestamps must adapt to the interval between measurements. Keys are ISO8601 intervals.

defaultIndex

Default is null. This property specifies the default index.

metaFields

An array of fields outside of _source. Kibana merges these fields into the document when displaying the document.

discover:sampleSize

The number of rows to show in the Discover table.

doc_table:highlight

Highlight results in Discover and Saved Searches Dashboard. Highlighting makes request slow when working on big documents. Set this property to false to disable highlighting.

courier:maxSegmentCount

Kibana splits requests in the Discover app into segments to limit the size of requests sent to the Elasticsearch cluster. This setting constrains the length of the segment list. Long segment lists can significantly increase request processing time.

fields:popularLimit

This setting governs how many of the top most popular fields are shown.

histogram:barTarget

When date histograms use the auto interval, Kibana attempts to generate this number of bars.

histogram:maxBars

Date histograms are not generated with more bars than the value of this property, scaling values when necessary.

visualization:tileMap:maxPrecision

The maximum geoHash precision displayed on tile maps: 7 is high, 10 is very high, 12 is the maximum. Explanation of cell dimensions.

visualization:tileMap:WMSdefaults

Default properties for the WMS map server support in the tile map.

visualization:colorMapping

Maps values to specified colors within visualizations.

visualization:loadingDelay

Time to wait before dimming visualizations during query.

csv:separator

A string that serves as the separator for exported values.

csv:quoteValues

Set this property to true to quote exported values.

history:limit

In fields that have history, such as query inputs, the value of this property limits how many recent values are shown.

shortDots:enable

Set this property to true to shorten long field names in visualizations. For example, instead of foo.bar.baz, show f.b.baz.

truncate:maxHeight

This property specifies the maximum height that a cell occupies in a table. A value of 0 disables truncation.

indexPattern:fieldMapping:lookBack

The value of this property sets the number of recent matching patterns to query the field mapping for index patterns with names that contain timestamps.

format:defaultTypeMap

A map of the default format name for each field type. Field types that are not explicitly mentioned use "default".

format:number:defaultPattern

Default numeral format for the "number" format.

format:bytes:defaultPattern

Default numeral format for the "bytes" format.

format:percent:defaultPattern

Default numeral format for the "percent" format.

format:currency:defaultPattern

Default numeral format for the "currency" format.

timepicker:timeDefaults

The default time filter selection.

timepicker:refreshIntervalDefaults

The time filter’s default refresh interval.

dashboard:defaultDarkTheme

Set this property to true to make new dashboards use the dark theme by default.

Kibi Specific Settings Reference
kibi:awesomeDemoMode

Set to true to suppress all warnings and errors.

kibi:splitTabs

Set to true to split dashboard tabs on two lines.

kibi:timePrecision

Set to generate time filters with certain precision. Possible values are: s, m, h, d, w, M, y.

kibi:relationalPanel

Display the Relational Panel in the dashboard tab.

kibi:relations

Relations between index patterns and dashboards.

kibi:session_cookie_expire

Set duration of cookie session (in seconds).

kibi:enableAllDashboardsCounts

Enable counts on all dashboards.

kibi:enableAllRelBtnCounts

Enable counts on all relational buttons.

Setting Kibi Server Properties

The Kibi server reads properties from the kibi.yml file on startup. The default settings configure Kibana to listen on port 5606 on all the IP addresses. To change the host or port number, or connect to Elasticsearch running on a different machine, you’ll need to update your kibi.yml file. You can also enable SSL and set a variety of other options.

deprecated[4.2, The names of several Kibana server properties changed in the 4.2 release of Kibana. The previous names remain as functional aliases, but are now deprecated and will be removed in a future release of Kibana]

Kibana Server Properties Changed in the 4.2 Release
server.port added[4.2]

The port that the Kibana server runs on.

alias: port deprecated[4.2]

default: 5606

server.host added[4.2]

The host to bind the Kibana server to.

alias: host deprecated[4.2]

default: "0.0.0.0"

elasticsearch.url added[4.2]

The Elasticsearch instance where the indices you want to query reside.

alias: elasticsearch_url deprecated[4.2]

default: "http://localhost:9200"

elasticsearch.preserveHost added[4.2]

By default, the host specified in the incoming request from the browser is specified as the host in the corresponding request Kibana sends to Elasticsearch. If you set this option to false, Kibana uses the host specified in elasticsearch_url.

alias: elasticsearch_preserve_host deprecated[4.2]

default: true

elasticsearch.ssl.cert added[4.2]

This parameter specifies the path to the SSL certificate for Elasticsearch instances that require a client certificate.

alias: kibana_elasticsearch_client_crt deprecated[4.2]

elasticsearch.ssl.key added[4.2]

This parameter specifies the path to the SSL key for Elasticsearch instances that require a client key.

alias: kibana_elasticsearch_client_key deprecated[4.2]

elasticsearch.password added[4.2]

This parameter specifies the password for Elasticsearch instances that use HTTP basic authentication. Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server.

alias: kibana_elasticsearch_password deprecated[4.2]

elasticsearch.username added[4.2]

This parameter specifies the username for Elasticsearch instances that use HTTP basic authentication. Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server.

alias: kibana_elasticsearch_username deprecated[4.2]

elasticsearch.pingTimeout added[4.2]

This parameter specifies the maximum wait time in milliseconds for ping responses by Elasticsearch.

alias: ping_timeout deprecated[4.2]

default: 1500

elasticsearch.startupTimeout added[4.2]

This parameter specifies the maximum wait time in milliseconds for Elasticsearch discovery at Kibana startup. Kibana repeats attempts to discover an Elasticsearch cluster after the specified time elapses.

alias: startup_timeout deprecated[4.2]

default: 5000

kibana.index added[4.2]

The name of the index where saved searched, visualizations, and dashboards will be stored..

alias: kibana_index deprecated[4.2]

default: .kibana

kibana.defaultAppId added[4.2]

The page that will be displayed when you launch Kibana: discover, visualize, dashboard, or settings.

alias: default_app_id deprecated[4.2]

default: "discover"

logging.silent added[4.2]

Set this value to true to suppress all logging output.

default: false

logging.quiet added[4.2]

Set this value to true to suppress all logging output except for log messages tagged error, fatal, or Hapi.js errors.

default: false

logging.verbose added[4.2]

Set this value to true to log all events, including system usage information and all requests.

default: false

logging.events added[4.2]

You can specify a map of log types to output tags for this parameter to create a customized set of loggable events, as in the following example:

{
  log: ['info', 'warning', 'error', 'fatal'],
  response: '*',
  error: '*'
}
elasticsearch.requestTimeout added[4.2]

How long to wait for responses from the Kibana backend or Elasticsearch, in milliseconds.

alias: request_timeout deprecated[4.2]

default: 500000

elasticsearch.shardTimeout added[4.2]

How long Elasticsearch should wait for responses from shards. Set to 0 to disable.

alias: shard_timeout deprecated[4.2]

default: 0

elasticsearch.ssl.verify added[4.2]

Indicates whether or not to validate the Elasticsearch SSL certificate. Set to false to disable SSL verification.

alias: verify_ssl deprecated[4.2]

default: true

elasticsearch.ssl.ca

An array of paths to the CA certificates for your Elasticsearch instance. Specify if you are using a self-signed certificate so the certificate can be verified. Disable elasticsearch.ssl.verify otherwise.

alias: ca deprecated[4.2]

server.ssl.key added[4.2]

The path to your Kibana server’s key file. Must be set to encrypt communications between the browser and Kibana.

alias: ssl_key_file deprecated[4.2]

server.ssl.cert added[4.2]

The path to your Kibana server’s certificate file. Must be set to encrypt communications between the browser and Kibana.

alias: ssl_cert_file deprecated[4.2]

pid.file added[4.2]

The location where you want to store the process ID file.

alias: pid_file deprecated[4.2]

default: /var/run/kibana.pid

logging.dest added[4.2]

The location where you want to store the Kibana’s log output. If not specified, log output is written to standard output and not stored. Specifying a log file suppresses log writes to standard output.

alias: log_file deprecated[4.2]

Managing Saved Searches, Visualizations, and Dashboards

You can view, edit, and delete saved searches, visualizations, and dashboards from Settings > Objects. You can also: * export or import sets of objects (searches, visualizations, dashboards, etc.) by using "Export" button * export all objects plus configuration, and index-patterns by using "Export Everything" button

Viewing a saved object displays the selected item in the Discover, Visualize, or Dashboard page. To view a saved object:

  1. Go to Settings > Objects.

  2. Select the object you want to view.

  3. Click the View button.

Editing a saved object enables you to directly modify the object definition. You can change the name of the object, add a description, and modify the JSON that defines the object’s properties.

If you attempt to access an object whose index has been deleted, Kibana displays its Edit Object page. You can:

  • Recreate the index so you can continue using the object.

  • Delete the object and recreate it using a different index.

  • Change the index name referenced in the object’s kibanaSavedObjectMeta.searchSourceJSON to point to an existing index pattern. This is useful if the index you were working with has been renamed.

Warning
No validation is performed for object properties. Submitting invalid changes will render the object unusable. Generally, you should use the Discover, Visualize, or Dashboard pages to create new objects instead of directly editing existing ones.

To edit a saved object:

  1. Go to Settings > Objects.

  2. Select the object you want to edit.

  3. Click the Edit button.

  4. Make your changes to the object definition.

  5. Click the Save Object button.

To delete a saved object:

  1. Go to Settings > Objects.

  2. Select the object you want to delete.

  3. Click the Delete button.

  4. Confirm that you really want to delete the object.

To export a set of objects:

  1. Go to Settings > Objects.

  2. Select the type of object you want to export. You can export a set of objects (searches, visualizations, dashboards, etc) by using "Export" button or you can export all objects plus configuration, and index-patterns by using "Export Everything" button.

  3. Click the selection box for the objects you want to export, or click the Select All box.

  4. Click Export to select a location to write the exported JSON.

Important
If you need to export every object you have in your Kibi installation, you can use "Export Everything" button.
Warning
Exported dashboards do not include their associated index patterns (not "Export Everything" button). Re-create the index patterns manually before importing saved dashboards to a Kibana instance running on another Elasticsearch cluster.

To import a set of objects:

  1. Go to Settings > Objects.

  2. Click Import to navigate to the JSON file representing the set of objects to import.

  3. Click Open after selecting the JSON file.

  4. If any objects in the set would overwrite objects already present in Kibi, confirm the overwrite.

Kibi server configuration

The Kibi server reads its configuration from the config/kibi.yml file at startup. The default settings configure Kibi to run on http://localhost:5606. To change the host or port number, or connect to Elasticsearch running on a different machine, you’ll need to update your kibi.yml file. You can also enable SSL and set a variety of other options.

Environment variable placeholders

It is possible to use environment variable placeholders in configuration settings; the syntax of placeholders is ${ENV_VARIABLE_NAME}.

For example, to set elasticsearch.url to the value of the environment variable ES_URL, edit config_kibi.yml as follows:

elasticsearch.url: ${ES_URL}
Kibi Configuration Settings
server.port:

Default: 5606 Kibi is served by a back end server. This setting specifies the port to use.

server.host:

Default: "0.0.0.0" This setting specifies the IP address of the back end server.

server.basePath:

Enables you to specify a path to mount Kibi at if you are running behind a proxy. This setting cannot end in a slash (/).

server.maxPayloadBytes:

Default: 1048576 The maximum payload size in bytes for incoming server requests.

elasticsearch.url:

Default: "http://localhost:9200" The URL of the Elasticsearch instance to use for all your queries.

kibana.index:

Default: ".kibana" Kibi uses an index in Elasticsearch to store saved searches, visualizations and dashboards. Kibi creates a new index if the index doesn’t already exist.

kibana.defaultAppId:

Default: "discover" The default application to load.

tilemap.url:

Default: "https://tiles.elastic.co/v1/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana" The URL to the tile service that Kibi uses to display map tiles in tilemap visualizations. added[4.5.3]

tilemap.options.minZoom:

Default: 1 The minimum zoom level. added[4.5.3]

tilemap.options.maxZoom:

Default: 10 The maximum zoom level. added[4.5.3]

tilemap.options.attribution:

Default: "© [Elastic Tile Service](https://www.elastic.co/elastic-tile-service)" The map attribution string. added[4.5.3]

tilemap.options.subdomains:

An array of subdomains used by the tile service. Specify the position of the subdomain the URL with the token {s}.added[4.5.3]

elasticsearch.username: and elasticsearch.password:

If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Kibi server uses to perform maintenance on the Kibi index at startup. Your Kibi users still need to authenticate with Elasticsearch, which is proxied through the Kibi server.

server.ssl.cert: and server.ssl.key:

Paths to the PEM-format SSL certificate and SSL key files, respectively. These files enable SSL for outgoing requests from the Kibi server to the browser.

elasticsearch.ssl.cert: and elasticsearch.ssl.key:

Optional settings that provide the paths to the PEM-format SSL certificate and key files. These files validate that your Elasticsearch backend uses the same key files.

elasticsearch.ssl.ca:

Optional setting that enables you to specify a path to the PEM file for the certificate authority for your Elasticsearch instance.

elasticsearch.ssl.verify:

Default: true To disregard the validity of SSL certificates, change this setting’s value to false.

elasticsearch.pingTimeout:

Default: the value of the elasticsearch.requestTimeout setting Time in milliseconds to wait for Elasticsearch to respond to pings.

elasticsearch.requestTimeout:

Default: 30000 Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.

elasticsearch.shardTimeout:

Default: 0 Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

elasticsearch.startupTimeout:

Default: 5000 Time in milliseconds to wait for Elasticsearch at Kibi startup before retrying.

pid.file:

Specifies the path where Kibi creates the process ID file.

logging.dest:

Default: stdout Enables you specify a file where Kibi stores log output.

logging.filter.<key>:

Default: authorization Replace <key> with the string to filter. Set the value of this setting to remove to remove matching keys from all logged objects. Set the value of this setting to censor to replace each character in the key’s value with an X character.

logging.silent:

Default: false Set the value of this setting to true to suppress all logging output.

logging.quiet:

Default: false Set the value of this setting to true to suppress all logging output other than error messages.

logging.verbose

Default: false Set the value of this setting to true to log all events, including system usage information and all requests.

status.allowAnonymous

Default: false If authentication is enabled, setting this to true allows unauthenticated users to access the Kibana server status API and status page.

kibi_core.default_dashboard_title

Default: not set The dashboard that is displayed when clicking on the Dashboard tab for the first time.

External datasource configuration is documented in the External datasources chapter.

deprecated[0.3, The names of several Kibi server properties changed in the 0.3 release of Kibana. The previous names remain as functional aliases, but are now deprecated and will be removed in a future release of Kibi]

Kibi Server Properties Changed in the 0.3 Release
server.port added[0.3]

The port that the Kibana server runs on.

alias: port deprecated[0.3]

default: 5606

server.host added[0.3]

The host to bind the Kibana server to.

alias: host deprecated[0.3]

default: "0.0.0.0"

elasticsearch.url added[0.3]

The Elasticsearch instance where the indices you want to query reside.

alias: elasticsearch_url deprecated[0.3]

default: "http://localhost:9200"

elasticsearch.preserveHost added[0.3]

By default, the host specified in the incoming request from the browser is specified as the host in the corresponding request Kibana sends to Elasticsearch. If you set this option to false, Kibana uses the host specified in elasticsearch_url.

alias: elasticsearch_preserve_host deprecated[0.3]

default: true

elasticsearch.ssl.cert added[0.3]

This parameter specifies the path to the SSL certificate for Elasticsearch instances that require a client certificate.

alias: kibana_elasticsearch_client_crt deprecated[0.3]

elasticsearch.ssl.key added[0.3]

This parameter specifies the path to the SSL key for Elasticsearch instances that require a client key.

alias: kibana_elasticsearch_client_key deprecated[0.3]

elasticsearch.password added[0.3]

This parameter specifies the password for Elasticsearch instances that use HTTP basic authentication. Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server.

alias: kibana_elasticsearch_password deprecated[0.3]

elasticsearch.username added[0.3]

This parameter specifies the username for Elasticsearch instances that use HTTP basic authentication. Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server.

alias: kibana_elasticsearch_username deprecated[0.3]

elasticsearch.pingTimeout added[0.3]

This parameter specifies the maximum wait time in milliseconds for ping responses by Elasticsearch.

alias: ping_timeout deprecated[0.3]

default: 1500

elasticsearch.startupTimeout added[0.3]

This parameter specifies the maximum wait time in milliseconds for Elasticsearch discovery at Kibana startup. Kibana repeats attempts to discover an Elasticsearch cluster after the specified time elapses.

alias: startup_timeout deprecated[0.3]

default: 5000

kibana.index added[0.3]

The name of the index where saved searched, visualizations, and dashboards will be stored..

alias: kibana_index deprecated[0.3]

default: .kibana

kibana.defaultAppId added[0.3]

The page that will be displayed when you launch Kibana: discover, visualize, dashboard, or settings.

alias: default_app_id deprecated[0.3]

default: "discover"

logging.silent added[0.3]

Set this value to true to suppress all logging output.

default: false

logging.quiet added[0.3]

Set this value to true to suppress all logging output except for log messages tagged error, fatal, or Hapi.js errors.

default: false

logging.verbose added[0.3]

Set this value to true to log all events, including system usage information and all requests.

default: false

logging.events added[0.3]

You can specify a map of log types to output tags for this parameter to create a customized set of loggable events, as in the following example:

{
  log: ['info', 'warning', 'error', 'fatal'],
  response: '*',
  error: '*'
}
elasticsearch.requestTimeout added[0.3]

How long to wait for responses from the Kibana backend or Elasticsearch, in milliseconds.

alias: request_timeout deprecated[0.3]

default: 500000

elasticsearch.shardTimeout added[0.3]

How long Elasticsearch should wait for responses from shards. Set to 0 to disable.

alias: shard_timeout deprecated[0.3]

default: 0

elasticsearch.ssl.verify added[0.3]

Indicates whether or not to validate the Elasticsearch SSL certificate. Set to false to disable SSL verification.

alias: verify_ssl deprecated[0.3]

default: true

elasticsearch.ssl.ca

An array of paths to the CA certificates for your Elasticsearch instance. Specify if you are using a self-signed certificate so the certificate can be verified. Disable elasticsearch.ssl.verify otherwise.

alias: ca deprecated[0.3]

server.ssl.key added[0.3]

The path to your Kibana server’s key file. Must be set to encrypt communications between the browser and Kibana.

alias: ssl_key_file deprecated[0.3]

server.ssl.cert added[0.3]

The path to your Kibana server’s certificate file. Must be set to encrypt communications between the browser and Kibana.

alias: ssl_cert_file deprecated[0.3]

pid.file added[0.3]

The location where you want to store the process ID file.

alias: pid_file deprecated[0.3]

default: /var/run/kibana.pid

logging.dest added[0.3]

The location where you want to store the Kibana’s log output. If not specified, log output is written to standard output and not stored. Specifying a log file suppresses log writes to standard output.

alias: log_file deprecated[0.3]

Authentication and access control [Enterprise Edition only]

NOTE: Documentation for Authentication and access control is available only in Kibi Enterprise Edition.