Configuring the off-heap memory
To prepare your system for use, you must first configure the off-heap memory. The memory module is responsible for allocating and managing chunks of off-heap memory. For more information, see the Memory section of the Modules topic.
Checking off-heap memory allocation
Siren Federate provides a cluster-level API that allows you to retrieve statistics about the cluster and the off-heap memory allocation.
The allocated direct memory represents the off-heap memory chunks pre-allocated to accommodate the root allocator. The chunk of off-heap memory that was allocated is kept and reused, because off-heap memory allocation is expensive. The root allocator can then efficiently allocate off-heap memory buffers of various sizes.
Setting off-heap memory
You configure the amount of off-heap memory available for the root allocator by updating the siren.memory.root.limit
variable in the config/elasticsearch.yml
file. By default if the JVM option MaxDirectMemorySize
is unset, Elasticsearch allocates half of the heap memory for off-heap, i.e., the JVM option MaxDirectMemorySize
is half of MaxHeapSize
.
The settings |
To set a limit for the root allocator, complete the following steps:
-
In
config/elasticsearch.yml
, set thesiren.memory.root.limit
parameter to a value, for example,2G
.siren.memory.root.limit: 2G
This memory limit must be less than the
MaxDirectMemorySize
value. The default value ofMaxDirectMemorySize
is half of the-Xmx
if you do not set it explicitly. -
In
config/jvm.options
, set the-Xmx
parameter to at least twice the value you set in step 1, for example, 4 GB.-Xmx4g
The max heap will be 4 GB, and the
MaxDirectMemorySize
will be automatically set to2147483648
(2 GB), so you can put at most2147483647
insiren.memory.root.limit
It is forbidden to use a limit that is greater than or equal to the maximum direct memory limit.
-
Start the Elasticsearch instance. The following info logs are displayed as follows:
[2019-12-10T17:29:11,207][INFO ][i.s.f.c.i.m.BufferAllocatorService] [node_s0] Buffer allocator service starting with Unsafe access: true [2019-12-10T17:29:11,207][INFO ][i.s.f.c.i.m.BufferAllocatorService] [node_s0] Buffer allocator service starting with maxDirectMemory=2254438400 (1) [2019-12-10T17:29:11,233][INFO ][i.s.f.c.i.m.BufferAllocatorService] [node_s0] Buffer allocator service starting with defaultNumDirectArenas=5 [2019-12-10T17:29:11,236][INFO ][i.s.f.c.i.m.BufferAllocatorService] [node_s0] Instantiating root allocator with limit=2147483648 (2)
These info logs provide an overview of how Siren Federate is configured. You can see that:
1 The maximum direct memory limit is correctly set to the platform dependent memory limit. 2 The root allocator limit is correctly set to 2G
.
Recommended settings
It is critical to ensure that there is enough available memory on the machine to accommodate the maximum direct memory limit for Siren Federate, the JVM maximum heap memory limit, and the operating system.
If the sum of maximum direct memory limit for Siren Federate and the JVM maximum heap memory limit does not leave enough memory for the operating system, the OS might stop the Elasticsearch instance (OOM killer process on Linux systems). |
Configuring a 64 GB machine
The following are the recommended settings for a cluster that needs to execute joins on a large amount of data:
-
24 GB heap for Elasticsearch
-
12 GB off-heap for Siren Federate
-Xmx24g
siren.memory.root.limit: 12G
If the off-heap memory assigned to Siren Federate is not enough, you can increase it while trying to maintain at least a 1:30 ratio between free system memory and the size of local Elasticsearch data.
If the ratio between free system memory and the local Elasticsearch data is lower than 1:30 or you experience performance issues in search requests not using joins, it is recommended to start increasing the number of data nodes in the cluster to maximize the use of the filesystem cache.
Configuring a 128 GB machine
-
30 GB heap for Elasticsearch
-
15 GB off-heap for Federate
-Xmx30g
siren.memory.root.limit: 15G
Alternatively, if the off-heap memory for Siren Federate is not fully used, it is better to give more heap memory to the operating system and the OS cache.