Editing entity tables

In the Data model app, all entity tables are listed in a list to the left of the screen.

When an entity table is viewed in the Data model app, its settings are divided into several tabs. These tabs are where you can edit an entity table’s structure, view its data, and update its configuration options.

You can perform the following tasks:

The Info tab

On the Info tab, you can modify the basic information about the entity table, such as the name, icon, and color.

You can also set the following properties:

  • Default label: You can compose the label that is used for the records when they are viewed in the Graph Browser, the record view, or in global search results. This can be either a Scripted Label or a Document Field. If you are using a scripted label, you must use a specific syntax to display the label. For example, @doc[_source][title]@ displays the contents of the title field as the label.

  • Default image: In a similar way to the default label, you can specify an image that displays for the entity table, either by using a Scripted Label or a Document Field. If you are using a scripted label, you must use a specific syntax to display the label. For example, use syntax like the following to add the contents of the file_name field to the url: http://example.com/datafiles/@doc[_source][file_name]@.

  • Index pattern used by this table: Specify an index pattern that names the Elasticsearch indices that are used by this entity table. You can update the index pattern to change its associated index or even to select multiple indices.

    To specify multiple indices, you can add either an asterisk (*) or a comma (,). For example, the pattern myindex-* matches all indices with names that start with myindex-, such as myindex, myindex-1, and myindex-2. The pattern myindex-1,myindex-2 includes myindex-1 and myindex-2, but not myindex.

    Changing this setting will impact this entity table, the searches derived from it, and all of the associated dashboards and visualizations.

  • Time filter: Select Enable the Time filter based on the following field if you want the entity table to be filterable by time. In the dropdown list that appears, specify a valid time field from the underlying Elasticsearch index.

The Fields tab

You can use the search field to search for specific fields in the entity table that you want to inspect or configure.

You can switch between Fingerprint view and Standard view: Fingerprint view displays some additional information about fields, such as samples, the number of non-null values, date range, and so on.

The fields are read-only, but can be modified by clicking Edit Mode.

In edit mode, you can add, rename, and delete fields.

If the number of records in the entity table exceeds 10,000, then editing fields is restricted because it would require a re-indexing of the index. If you need to increase the threshold of the entity table, go to Management → Advanced Settings and modify the data-model:max_documents_for_reindex_operations parameter.

You can also activate the following options:

  • Unique value: Select the check box on fields that contain unique values in each record. For example, if your data contains an ID field, the value of the ID can help to identify that record in the UI.

  • Single value: Select the check box on fields that contain no more than one value.

  • Format: Allows you to control the way that specific values are displayed. It can also cause values to be completely changed and prevent highlighting in Discover from working. The following options are available:

    • Boolean

    • color

    • string (default)

    • truncated string

    • URL

The following information is available in columns for each of the fields:

  • Searchable: Indicates that the field can be searched by using the search bar or the global search interface.

  • Aggregatable: Indicates that the field can be used in aggregations in visualizations.

  • Excluded: Indicates that the field is excluded from _source when it is fetched.

  • Popularity: Modify this value to change the order in which record contents are displayed.

If your index is based on a star pattern or is a multi-index, you cannot add, rename, or delete fields.

You can sort the fields by clicking their column header. And you can switch to ascending or descending order by clicking the arrow that appears in the column header that you sort by.

Refresh the field table by clicking Refresh field list.

The Data tab

From the Data tab, you can explore the data in the entity table, which can be useful after the data is first imported.

You can also perform the following actions:

  • Generate dashboards for the entity table. For more information, see Auto-generating dashboards.

  • Create a search. For more information, see Creating searches.

  • Create a record. If record editing is enabled, you can create new records in the entity table. For more information, see Setting up record editing.

  • Perform a bulk import of data from a spreadsheet or a datasource.

The Relations tab

The Relations tab allows you to define relations between entities.

For more information, see Adding relations between entities.

The Scripted fields tab

Scripted fields are computed in real time from your data. They can be used in visualizations and displayed in your records. However, they cannot be searched.

Before you use scripted fields, see the Elasticsearch documentation about script fields and scripts in aggregations.

Scripted fields can be used to display and aggregate calculated values. For this reason, they can be very slow and, if configured incorrectly, they can cause Siren Investigate to become unusable. There is no protection from unexpected exceptions that are caused by script errors.

Scripted fields use the languages that are enabled in the back-end system. By default, the language that is used is the Painless Elasticsearch language.

Ensure that you access the version of Elasticsearch documentation that matches the back-end version that you are using.

To access values in the document, use the following format:

doc['some_field'].value

Painless is powerful and easy to use. It provides access to many native Java APIs and has an easy-to-learn syntax.

Currently, Siren Investigate does not support named functions in Painless scripts. Alternatively, you can use Lucene Expressions. These are a lot like JavaScript, but limited to basic arithmetic, bitwise and comparison operations.

Lucene Expressions have the following limitations:

  • Only numeric, boolean, date and geo_point fields may be accessed.

  • Stored fields are not available.

  • If a field is sparse (only some documents contain a value), documents missing the field will have a value of 0.

Table 1. Supported operators and functions in Lucene xpressions

Arithmetic operators

+, -, *, /, %

Bitwise operators

, &, ^, ~, <<, >>, >>>

Boolean operators* (including the ternary operator)

&&, !, ?:

Comparison operators

<, ⇐, ==, >=, >

Common mathematic functions

abs, ceil, exp, floor, ln, log10, logn, max, min, sqrt, pow

Trigonometric library functions

acosh, acos, asinh, asin, atanh, atan, atan2, cosh, cos, sinh, sin, tanh, tan

Distance functions

haversin

Miscellaneous functions

min, max

*Another boolean operator that is supported is ||

Scripted fields have the following properties:

  • Name

  • Language:

    • expression

    • painless (default)

  • Type:

    • Boolean

    • date

    • number

    • string

  • Format: Enables you to control the way that specific values are displayed. It can also cause values to be completely changed and prevent highlighting in Discover from working. The following options are available:

    • Boolean

    • bytes

    • color

    • duration

    • number (default)

    • percentage

    • string

    • URL

  • Popularity

  • Unique value

  • Single value

  • Script

Click Create field to save your changes.

The Options tab

On the Options tab, you can take measures to filter data to improve system performance.

Source filters

Source filters can be used to exclude one or more fields when fetching the document source. This happens when viewing a document in the Discover app, or with a table displaying results from a saved search in the Dashboard app.

Each row is built using the source of a single record. If you have records with large or unimportant fields you may benefit from filtering those out at this lower level.

Multi-fields will incorrectly appear as matches in the table. These filters only apply to fields in the original source document, so that matching multi-fields are actually not filtered.

Enter a string in the Source Filter field and click Add. Filters accept wildcards, for example, user* will return fields starting with "user".

Sampling on the graph

In the Graph Browser, you can set a graph expansion limit, which controls how many records can be imported into a graph from a dashboard. This is called sampling.

For more information, see Sampling data in the graph.

Minimizing expensive queries

When using dashboards sometimes removing a filter or setting a broad time range can cause the system to produce queries which are too expensive to process. To prevent this situation, you can set limits on the time range that’s applied, or on the number of records that can be added to a dashboard.

For more information, see Minimizing expensive queries.

Highlighting

Siren Investigate applies Lucene highlighter methods to compute how highlighting is applied to the text.

In the advanced settings, the setting default:highlight-type is available, which is set to UNIFIED by default. The other available highlight type is PLAIN. For more information, see the Elasticsearch documentation.

The Template scripts tab

To enable this feature, make sure that scripting is enabled and that the React library is added to the librariesWhitelist property in investigate.yml. Optionally, EUI can also be added if you wish to use EUI based components.

siren_scripting:
  enabled: true
  librariesWhitelist:
    - React
    - EUI

The Template scripts tab gives you the ability to add scripts for creating custom views in the Overview tab of the Record View. Template scripts can also define actions to download the record view in any file format.

You can add any compatible existing sirenapi script by clicking the Add scripts button and selecting your preferred scripts. The Overview tab will appear in the Record View as long as one template script is defined in the list.

A preview of each template is available in the panel to the right. Click any template from your selected templates list to preview it. When viewing all templates in the 'Add template scripts' modal, you can preview any of these templates by clicking the play button displayed alongside each one.

You can create a new template script by clicking the "Create Script" button or make changes to a template script by clicking the edit button. This can be done before or after a script is added to the entity table.

See the following example about how to create your template script:

const version = 1;
const type = "template";

/**
 * Code to fetch relevant record data.
 * Check the DataRecord, DataModelEntity and Relation classes in the SirenAPI documentation for more details.
 */
async function fetchRecordData(record, search) {
  const label = await record.getLabel();
  const fieldValues = await record.getFormattedFieldValues('field');

  // ...

  return { label, fieldValues, /* ... */ };
}

/**
 * View function for the Record Viewer.
 *
 * This function will receive a DataRecord instance corresponding to the record itself and a DataModelEntity instance
 * for the Search entity the Record Viewer is spawned from. The input Search will always contain the input Record.
 *
 * The third additional parameter is a render function that can be used to display partial results about the record.
 *
 * The returned value is a React element written using the JSX syntax that will be rendered in the
 * Record View > Overview tab.
 */
async function buildReactRecordView(record, search, render) {
  render(<h1>Hello world is loading...</h1>);

  const recordData = await fetchRecordData(record, search);
  return <h1>Hello World, {recordData.label}!</h1>;
}

/**
 * Alternative record view using JsReport templates exposed in the Siren API.
 * Note that you need to know the JsReport template id to be rendered with the calculated record data.
 */
async function buildJsReportRecordView(record, search) {
  const recordData = await fetchRecordData(record, search);
  return sirenapi.Reporting.renderToElement(htmlTemplateId, recordData); // htmlTemplateId is the target template id
}

/**
 * Download functions for different file types.
 *
 * These functions will also receive a record and an associated DataModelEntity, but they will trigger
 * a file download of their specific file type.
 *
 * The download should typically happen through the JsReport proxy exposed in the Siren API.
 */
async function buildPdfDownload(record, search) {
  const recordData = await fetchRecordData(record, search);
  sirenapi.Reporting.download(pdfTemplateId, recordData, 'record.pdf');
}

// The registered object lists all available views/downloads for this template.
context.registerTemplate({
  recordView: buildReactRecordView,

  download: {
    pdf: buildPdfDownload,
    // ...More file types can be added as required
  }
});

For more information about script editing, see Creating scripts.

The Editing tab

The settings on the Editing tab allow you to configure how changes made to a record of an entity table should be handled.

For more information, see Setting up record editing.

The Searching tab

The Searching tab contains settings that you can use to control the results that appear in the Global Search interface.

To enable searching on the entity table, select the checkbox labeled The <name> entity table is available in searches. Only entity tables that are enabled for searching will appear in the Search in dropdown menu of the Global Search and in Global Search results.

You can set the Boost search value of an entity table to prioritize its appearance in search results over other entity tables. The boost value multiplies the search scores of all records of the entity table, making them more likely to appear first in search results.

The boost value of an entity table is actually applied on the associated Elasticsearch indices. In situations where an index is included in multiple entity tables, the boost level will be taken from the entity table with the greater value.

For example, if one entity table has an index pattern index1,index2 with a boost value of 7.0 and another entity table has an index pattern index2,index3 with a boost value of 8.0, the boost value for index2 will be set at 8.0.

You can enable or disable and set the boost value for individual fields in the Field settings section. To enable or disable fields for searching, select the check boxes under the Searchable field column. To change field boost values, edit the text boxes in the Search boost column.

When running queries, Elasticsearch does not provide a way to differentiate between fields with the same name in different indices. As a consequence, if you have the same field in two separate entity tables, then it will be considered "enabled" for searching when it is enabled in at least one of the entity tables. Similarly, the applied boost value will be the maximum boost value among all entity tables where it is enabled for searching.

The Data model graph

On the Data model graph tab, you can visualize the connected data in a graph to see how the entities relate to one another.

The graph gives a visual representation of the relations (the connections between entities) that are configured on the Relations tab.

image

Deleting entity tables

To delete an entity table, select the entity table in the list and click Delete in the Options menu.

Select one of following options:

  • Remove entity table object only: Removes the entity table from the data model and retains the data in the index.

  • Delete data only: Deletes the data from the index and retains the entity table object in the data model.

  • Delete all: Deletes the data from the index, deletes the index, and removes the entity table object from the data model.

The last two options are available only if the index is based on a single index or an index that is not a star pattern. If you select either of these options, you must confirm your selection by typing the word DELETE into a text field.

Refreshing the index fields list

When you add an index mapping, Siren Investigate automatically scans the indices that match the pattern to display a list of the index fields.

You can refresh the index fields list to pick up any newly-added fields.

Refreshing the index fields list also resets Siren Investigate’s popularity counters for the fields.

The popularity counters keep track of the fields you have used most often within Siren Investigate and are used to sort fields within lists.

To refresh the index fields list, go to the Fields tab and click Refresh field list.