Reconciliation API

This section focuses on the reconciliation API. For information about performing reconciliation, see Reconciliation.

Only administrators can run reconciliation.

Endpoint ⇒ POST /api/collections/reconciliation

Parameters

Parameter Type Required Description

entityTableIds

string[]

false

Reconcile document references from specific entity tables

indexPatterns

string[]

false

Reconcile document references from specific Elasticsearch index patterns

If neither entityTableIds or indexPatterns are provided, all document references will be reconciled.

You cannot use both entityTableIds and indexPatterns together - only one or the other.

Sample requests

A request to reconcile document references belonging to the 'company' index:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{ "indexPatterns": ["company"] }'

A request to reconcile document references belonging to the entity table with an ID of search:9b412ba0-533c-411e-b431-532db5d05260:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{ "entityTableIds": ["search:9b412ba0-533c-411e-b431-532db5d05260"] }'

A request to reconcile all document references:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{}'

Response

Reconciliation is a resource-intensive, long-running operation. Once user credentials are verified, Investigate will immediately respond with HTTP status code 202 (Accepted) and an operationId you can use to track its progress.

You should avoid starting multiple simultaneous reconciliation operations. Use the following operations API to wait for a reconciliation operation to complete before starting a new reconciliation.
Property Type Description

operationId

string

A UUID string that identifies the operation.

Sample response

{
  "operationId": "7e990a3a-9872-4157-9bea-6f7b6afe64d0"
}

Endpoint ⇒ GET /api/operations/{operationId}/stream

You can use this endpoint to receive a stream of notifications about the status of the operation represented by operationId. The stream follows the server-sent events specification.

The stream will emit the following events, representing a status object with data encoded as a JSON string:

Event Data type Description

update

ProgressData

Reports a progress update, including the overall completion percentage and a textual description of current objects being processed.

done

ReconciliationResult

Reports the final result of the reconciliation operation.

error

ErrorData

Reports an error in the procedure.

Sample request

curl -N -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/stream'

Sample response

event: update
data: {"text":"Update graphs 122/326","progressPercent":37}

event: update
data: {"text":"Update graphs 257/326","progressPercent":78}

id: end
event: done
data: {"totalRecordsCount":3000,"failedRecordsCount":0,"totalCollectionsCount":3,"failedCollectionsCount":0,"elapsedSeconds":3,"failedRecordsDetails":[],"failedCollectionsDetails":[]}

Endpoint ⇒ GET /api/operations/{operationId}/status

You can poll at this endpoint to get information about the current status of the long-running operation represented by operationId. The returned status object has the following schema:

Property Type Description

status

'running' | 'done' | 'error'

Current status of the operation.

data

ProgressData | ReconciliationResult | ErrorData

The data object associated with the current status.

Sample request

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/status'

Sample responses

{
  "status": "running",
  "data": {
    "text": "Update graphs 122/326",
    "progressPercent": 37
  }
}
{
  "status": "done",
  "data": {
    "totalRecordsCount": 3000,
    "failedRecordsCount": 0,
    "totalCollectionsCount": 3,
    "failedCollectionsCount": 0,
    "elapsedSeconds": 3,
    "failedRecordsDetails": [],
    "failedCollectionsDetails": []
  }
}
{
  "status": "error",
  "data": {
    "status": 400,
    "message": "Any error detail here"
  }
}

Status objects

ReconciliationResult

Representation of the result of a reconciliation operation.

Property Type Description

totalRecordsCount

number

Count of affected collection records

failedRecordsCount

number

Count of affected collection records that could not be reconciled

totalCollectionsCount

number

Count of collections with affected document references

failedCollectionsCount

number

Count of collections with affected document references that could not be reconciled

elapsedSeconds

number

Time taken for the reconciliation operation to run

failedRecordsDetails

string[]

Details of any records that could not be reconciled (first 100 entries)

failedCollectionsDetails

string[]

Details of any collections containing document references that could not be updated (first 100 entries)

ProgressData

An object representing current progress in the operation.

Property Type Description

text

string

Textual explanation of the current moment in the processing pipeline.

progressPercent

number

Completion percentage of the whole operation.

ErrorData

An object representing an error in the operation.

Property Type Description

status

number

HTTP status code representing the error.

message

string

Textual description of the error.