Elasticsearch REST API : useful commands

Introduction

This document is broken down into three sections:

• Generic Elasticsearch queries to retrieve details on the cluster/indices/statistics etc.
• Waratek event document/index format and some sample queries
• Instructions for purging event data from Elasticsearch

This document covers a very small subset of Elasticsearch queries/capabilities and assumes a basic knowledge of indices and documents. For a full reference, please go to the official Elasticsearch Guide (https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html).

All requests to Elasticsearch are done through the built-in REST API. The simplest way to run queries is via a command line tool like curl, however it is also recommended to use a GUI tool such as Postman, which will make maintaining/sharing/writing queries much simpler.

Authentication

All the queries below assume Elasticsearch is running on http with no authentication required. If using curl, specify a username/password using -u <user>:<password>. Add -k if running on https with a self-signed certificate. All the examples below assume Elasticsearch is running over http on the local system. If running over https please replace http with https in the commands.

Basic Elasticsearch administration

Note the Authentication section above before using these commands.

Ensuring Elasticsearch is up and accessible :

curl -XGET 'http://localhost:9200'

General health :

curl -XGET 'http://localhost:9200/_cat/health?v'

Disk usage (note : this lists disk space taken by Elasticsearch, plus total used disk space, in the example below Elasticsearch is using 1.1 GB, and total disk space used by the entire OS is 18.8 GB) :

es> curl -XGET "http://localhost:9200/_cat/allocation?v&pretty"
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
316 1.1gb 18.8gb 53.7gb 72.6gb 25 10.31.1.199 10.31.1.199 ip-10-31-1-199
316 UNASSIGNED
es>

List indices :

curl -XGET 'localhost:9200/_cat/indices?v'

There are many more cluster and node queries that can be run as needed. For more info, please go to Cluster APIs (https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html).

Delete a specific index :

curl -XDELETE 'http://localhost:9200/<index-name>'

Waratek security event queries

Security events generated by the Agents use an index-by-day pattern. The index pattern is in the format event-yyyy-MM-dd, so for example events generated on Mar 17 2021 would end up in the index event-2021-03-17.The index is created on demand so doesn’t exist until an indexing request is received.

A single event document is indexed in the format:

{
  "severity": "High",
  "mod": "Mod for filesystem controls",
  "modVersion": "2.3",
  "rule": "Allow read access to /etc directory rule",
  "eventType": "Execute Rule",
  "message": "CEF:0|ARMR:ARMR|ARMR|2.0|Allow read access to /etc directory rule|Execute Rule|High|act=allow msg=example of message outcome=success path=/etc/passwd",
  "triggeredDate": "2020-09-17 13:11:51.289 +0000",
  "nodeId": 1
}

A simple search to find all events in all event indices, ordered by triggeredDate in descending order

curl -XGET 'http://localhost:9200/event*/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
  "query": {
    "match_all": {}
  },
  "sort": {
    "triggeredDate": "desc"
  }
}'

Searches can run against all indices or a subset by using a wildcard. To search in a specific index replace the event* above with event-2021-03-17.

A more complicated query to search for all High severity events triggered in the last day: (note: eventType parameter filters out lifecycle events)

curl -XGET 'http://localhost:9200/event*/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "triggeredDate": {
              "gt": "now-1d/d",
              "lte": "now/d"
            }
          }
        }
      ],
      "must": [
        {
          "term": {
            "eventType.keyword": {
              "value": "Execute Rule"
            }
          }
        },
        {
          "term": {
            "severity.keyword": {
              "value": "High"
            }
          }
        }
      ]
    }
  }
}'

Deleting event indices

Proceed with caution when purging data from Elasticsearch

As with search, indices can be specified using a wildcard, here are some examples below To delete all event data :

curl -XDELETE 'http://localhost:9200/event*'

To delete all event data from 2020 :

curl -XDELETE 'http://localhost:9200/event-2020*'

To delete all event data between 1-Jan-2021 and 1-April-2021 :

curl -XDELETE 'http://localhost:9200/event-2020.01.*,event-2020.02.*,event-2020.03.*'