Elasticsearch : Requirements for different sized environments
Introduction
This document provides a summary of the environments (CPU/memory) required for different sized Elasticsearch installations. This is based on the number of events generated by Waratek Agent(s).
It should be noted that the Portal Dedicated itself and the database will need approximately 4GB of memory - so that needs to be added to the memory requirements for Elasticsearch below, to give a total memory requirement for the deployment server.
CPU/Memory Requirements
The metrics below were generated on a single Elasticsearch node
As a starting example, take the following scenario:
- Waratek Agent(s) are expected to generate 50,000 security events per minute
In this example - Waratek recommends deploying Elasticsearch on a 2-CPU system with 8GB memory.
The table below summarises the CPU/memory requirements for other customer scenarios, and can help planning infrastructure accordingly.
| Max events per minute / CPU cores (Memory) | 2 (8GB RAM) | 4 (16GB RAM) | 8 (32GB RAM) | 16 (64GB RAM) |
|---|---|---|---|---|
| 50,000 | ||||
| 100,000 | ||||
| 200,000 | ||||
| 400,000 | ||||
| 500,000 |
Storage Requirements
It is not possible to say exactly how much space each rule will take, especially if verbose logging (e.g. stack trace feature) is turned on, as each customer situation will differ.
However, Waratek have studied agents/events in a High Availability environment under load and can recommend having 1 GB disk space per 1 million events.
We do not recommend storing more than 150 million events on a single Elasticsearch node as the Portal Dedicated may hit timeouts querying Elasticsearch.