Migrating existing events from earlier versions

warning

This is only required if existing events need to be kept after upgrading the Portal Dedicated from versions v4.x.x or v5.x.x

• To maintain existing events it is required to migrate events if upgrading from any 4.x version.
• Migrating events is not required if upgrading from any 5.x version, however certain event fields will be missing if the events are not migrated

info

Migrating events is a slow process as each event needs to be updated and reindexed. As an approximation, for an averagely specced Elasticsearch cluster, it will take 10 minutes to migrate 1 million events.

The script to migrate existing events is migrateEvents.sh inside the elasticsearch directory of the new Portal Dedicated installation.

This script requires the four variables below to be set inside migrateEvents.sh prior to executing the script. These variables can be populated based on the values in application.properties.

info

Ensure Elasticsearch is running before executing this script.

# Elasticsearch username
ES_USER=""

# Elasticsearch password
ES_PASSWORD=""

# Elasticsearch hostname/IP
ES_HOST=""

# Elasticsearch port
ES_PORT=""

Once these values are set correctly - the script is executed without any flags:

cd elasticsearch
./migrateEvents.sh

producing output similar to:

Creating event pipeline
{"acknowledged":true}
Creating CEF pipeline parser
{"acknowledged":true}
Creating chain pipeline
{"acknowledged":true}
Creating trigger pipeline
{"acknowledged":true}
Creating ruleType pipeline
{"acknowledged":true}
Creating event template
{"acknowledged":true}
Adding ruleType mapping to existing indices
{"acknowledged":true}
Updating events
{"task":"Qkj6UIPbQ9KUj3ZM9nXGfQ:16379"}

The task to update events runs asynchronously.

To check the status of the migration use the Elasticsearch Task API endpoint, with the task ID output by the script above.

In the example above the task ID is Qkj6UIPbQ9KUj3ZM9nXGfQ:16379

so the task details can be retrieved via:

curl -X GET 'http://localhost:9200/_tasks/Qkj6UIPbQ9KUj3ZM9nXGfQ:13181'

replacing the Elasticsearch host details as appropriate.

The response will contain a completed flag which will be set to true when the migration is complete.

It is not required to wait for the task to complete before continuing, however events will only be displayed correctly after the migration is complete.

Done

All existing events have now been migrated so it is safe to continue with the upgrade.