Creating a Data Model using the Object Definitions API
Within this documentation, we will go over how to create a basic Data Model structure using the Object Definition API. In order to communicate with this API, we will use cURL. To learn more about this API, see Object Definitions API.
|
Throughout this tutorial, we will be calling this endpoint to create our objects:
However, if you would like to test your objects without ingesting them, you can utilize the Validate API by calling:
|
|
If you would like to delete objects that you have created, you can do so by utilising the Delete API:
|
This basic Data Model example will include the creation of: * Dataspace * Saved Searches. * Entity Identifiers (EIDs). * Relations. * Scripts.
Our Data Model will look like this. We have a Companies and People Entity Tables that contains relations to a Location Entity Identifier.
|
Within these examples, we will be sending requests to Investigate via For example:
The base path - |
1. Creating a Dataspace
We will be creating a Dataspace - test to host all of the objects we create. We create objects with the ?dataspace= query parameter. If you want to create all objects within your HOME dataspace, you can omit this query parameter and skip this step.
This will create an empty dataspace but it is possible to clone from other dataspaces by adding to the "options.copyFrom" array. For example, "copyFrom": ["HOME"] to clone the default dataspace.
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/dataspace" -d '
{
"options": {
"create": true,
"copyFrom": []
},
"attributes": {
"version": 100,
"shortCode": "test",
"title": "Test Dataspace",
"sharedWith": [{ "roleName": "<your-user-here>", "accessLevel": "dataspaceManager" }],
"icon": "fas fa-universal-access",
"colorCode": "#d20408"
}
}'2. Saved Searches
Before creating our Saved Search, we will create the index and mappings within Elasticsearch for its underlying data index. We will call it our people index add add some simple mappings:
curl -H "Content-Type: application/json" -XPUT "http://localhost:9220/people" -d '
{
"settings": {
"index": {
"number_of_replicas": 1,
"number_of_shards": 1
}
},
"mappings": {
"properties": {
"employer": {
"type": "keyword"
},
"first_name": {
"type": "keyword"
},
"last_name": {
"type": "keyword"
}
}
}
}'Now, we can create our Saved Search. As seen previously, we created mappings for three fields - employer, first_name and last_name. We will need to populate the "attributes.indexPattern.fields" property with a JSON stringified array of objects. These objects will represent fields and take the structure of:
{
"name": "<field-name>",
"label": "<field-label>",
"description": "<field-description>",
"esType": "<field-es-type>",
"type": "<field-type>",
"count": "<field-count>",
"primaryKey": "<primary-key-enabled>", // boolean
"singleValue": "<single-value-enabled>", // boolean
"scripted": "<scripted-enabled>", // boolean
"searchable": "<searchable-enabled>", // boolean
"aggregatable": "<aggregatable-enabled>", // boolean
"readFromDocValues": "<read-from-doc-vals-enabled>" // boolean
}If you would like to customise this "attributes.indexPattern.fields" array, create objects in this structure and JSON stringify them.
Create the People Saved Search:
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/search/search:ec984830-3007-11ec-a72f-7b66a29ade58?dataspace=test" -d '
{
"attributes": {
"title": "People",
"description": "",
"hits": 0,
"columns": [
"_source"
],
"sort": [],
"version": 100,
"indexPattern": {
"pattern": "people",
"fields": "[{\"name\":\"_id\",\"label\":\"_id\",\"description\":\"-\",\"esType\":\"_id\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"label\":\"_index\",\"description\":\"-\",\"esType\":\"_index\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"label\":\"_score\",\"description\":\"-\",\"esType\":\"text\",\"type\":\"number\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_siren\",\"label\":\"_siren\",\"description\":\"-\",\"esType\":\"object\",\"type\":\"unknown\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_siren.sic\",\"label\":\"_siren.sic\",\"description\":\"-\",\"esType\":\"object\",\"type\":\"unknown\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_siren.sic.namespace\",\"label\":\"_siren.sic.namespace\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_source\",\"label\":\"_source\",\"description\":\"-\",\"esType\":\"_source\",\"type\":\"_source\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"label\":\"_type\",\"description\":\"-\",\"esType\":\"_type\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"employer\",\"label\":\"employer\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"first_name\",\"label\":\"first_name\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"last_name\",\"label\":\"last_name\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]",
"excludeIndices": false
},
"siren": {
"indexingScope": "L",
"ui": {
"icon": "fas fa-user-tie",
"color": "#DE4D1F",
"shortDescription": ""
},
"globalSearch": {
"enabled": true,
"boost": 10
},
"templateScripts": {
"recordView": [],
"recordTable": []
},
"revision": {
"editableFieldsExceptions": [],
"editableFieldsMode": "blacklist",
"index": "people",
"indexingScope": "L",
"primaryKey": "_id",
"isEditable": true
}
},
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[{\"meta\":{\"id\":\"filter:revision-filter\",\"negate\":false,\"disabled\":false,\"alias\":\"Hide originals when revised\"},\"query\":{\"bool\":{\"must_not\":{\"term\":{\"_siren_revision.archived\":true}}}}}],\"highlightAll\":true,\"version\":true,\"query\":{\"match_all\":{}}}"
}
},
"isGlobal": false
}'Let’s create another Saved Search which will represent Companies. First, create the index and mappings before creating the Saved Search:
curl -H "Content-Type: application/json" -XPUT "http://localhost:9220/companies" -d '
{
"settings": {
"index": {
"number_of_replicas": 1,
"number_of_shards": 1
}
},
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"city": {
"type": "keyword"
}
}
}
}'Create the Companies Saved Search:
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/search/search:b61b87ab-0b9f-4f9b-90e8-2611c7e3c66e?dataspace=test" -d '
{
"attributes": {
"title": "Companies",
"description": "",
"hits": 0,
"columns": [
"_source"
],
"sort": [],
"version": 100,
"indexPattern": {
"pattern": "companies",
"fields": "[{\"name\":\"_id\",\"label\":\"_id\",\"description\":\"-\",\"esType\":\"_id\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"label\":\"_index\",\"description\":\"-\",\"esType\":\"_index\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"label\":\"_score\",\"description\":\"-\",\"esType\":\"text\",\"type\":\"number\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_siren\",\"label\":\"_siren\",\"description\":\"-\",\"esType\":\"object\",\"type\":\"unknown\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_siren.importId\",\"label\":\"_siren.importId\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_siren.importTimestamp\",\"label\":\"_siren.importTimestamp\",\"description\":\"-\",\"esType\":\"date\",\"type\":\"date\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_siren.importUser\",\"label\":\"_siren.importUser\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_siren.sic\",\"label\":\"_siren.sic\",\"description\":\"-\",\"esType\":\"object\",\"type\":\"unknown\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_siren.sic.namespace\",\"label\":\"_siren.sic.namespace\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_source\",\"label\":\"_source\",\"description\":\"-\",\"esType\":\"_source\",\"type\":\"_source\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"label\":\"_type\",\"description\":\"-\",\"esType\":\"_type\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"id\",\"label\":\"id\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"location\",\"label\":\"location\",\"description\":\"-\",\"esType\":\"keyword\",\"type\":\"string\",\"count\":0,\"primaryKey\":false,\"singleValue\":false,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]",
"excludeIndices": false
},
"siren": {
"indexingScope": "L",
"ui": {
"icon": "fas fa-building",
"color": "#1FDEBC",
"shortDescription": ""
},
"globalSearch": {
"enabled": true,
"boost": 10
},
"templateScripts": {
"recordView": [],
"recordTable": []
},
"revision": {
"editableFieldsExceptions": [],
"editableFieldsMode": "blacklist",
"index": "companies",
"indexingScope": "L",
"primaryKey": "_id",
"isEditable": true
}
},
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{ \"highlightAll\": true, \"query\": {}, \"filter\": [], \"version\": true }"
}
},
"isGlobal": false
}'|
You can also create a child Search using the "siren.parentId" property, after creating the Saved Search above, you will receive an |
3. Entity Identifiers (EIDs)
We will now create a Location EID and add it to our Data Model:
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/eid/eid:c6e0beb0-47e3-11ee-96b3-ab33c1b44c25?dataspace=test" -d '
{
"attributes": {
"title": "Location",
"version": 100,
"siren": {
"ui": {
"icon": "fas fa-map-marker-alt",
"color": "#FCB045",
"shortDescription": ""
}
}
}
}'4. Relations
Relations are used to associate our Saved Searches. Here, we will create a relation between our Companies and People Searches. The relation represents People (employer field) who work for Companies (id field).
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/relation?dataspace=test" -d '
{
"attributes": {
"title": "",
"version": 100,
"countEnabled": true,
"undirected": false,
"directLabel": "works for",
"domainField": "employer",
"domainId": "test:search:ec984830-3007-11ec-a72f-7b66a29ade58",
"inverseOf": "test:relation:b060f360-e386-11ed-a7f8-5960b7d66484",
"rangeField": "id",
"rangeId": "test:search:b61b87ab-0b9f-4f9b-90e8-2611c7e3c66e",
"inverseLabel": "employs",
"joinType": "",
"timeout": -1
},
"isGlobal": false
}'We can also create a relation between our Companies Saved Search and our created Location EID.
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/relation?dataspace=test" -d '
{
"attributes": {
"title": "companies -> has office in -> Location",
"version": 100,
"domainId": "test:search:b61b87ab-0b9f-4f9b-90e8-2611c7e3c66e",
"domainField": "location",
"rangeId": "test:eid:c6e0beb0-47e3-11ee-96b3-ab33c1b44c25",
"directLabel": "has office in",
"inverseLabel": "shares",
"inverseOf": "relation:f0f60f70-47e3-11ee-9f97-6ffb438a6c4d",
"timeout": -1,
"joinType": "",
"undirected": false,
"countEnabled": true
}
}'5. Siren API Scripts
For Siren API Scripts, you can pass in your script using the scriptSource parameter. To learn more, see
Scripting API
curl -H "Content-Type: application/json" -H "kbn-xsrf: any" -u sirenadmin:password -L -XPOST "http://localhost:5606/api/v1/object_definitions/sirenapiscript" -d '
{
"attributes": {
"title": "My Script",
"version": 100,
"apiVersion": 1,
"scriptTypeVersion": 1,
"scriptSource": "",
"type": "custom"
}
}'