Skip to main content
Version: 6.10.1

Troubleshooting

Users cannot connect to the Portal Dedicated server​

A browser on one computer cannot connect to the Portal Dedicated server running on a different computer

  • Ensure that the Portal Dedicated process is running

    $ ps -ef | grep java

    You should see the Portal Dedicated Java process is running

  • Open the logs

    $ tail -f /var/log/management-console/mc.log

    You should see no errors and that the process started correctly

  • Ensure the web application is accessible locally on the Portal Dedicated server

    $ curl -v -k https://localhost:8443

    You should see the Portal Dedicated HTML displayed on the screen

  • Ensure the Portal Dedicated is accessible over the network

    $ curl -v -k https://console_machine_ip_address:8443

  • If this fails to show the Portal Dedicated, set up port forwarding to connect to the Portal Dedicated server via port 8443

    $ ssh -L 8443:localhost:8443 console_machine_ip_address
    $ curl -v -k https://localhost:8443

    This should now succeed and display the HTML for the Portal Dedicated. Try using the browser to connect to the Portal via port forwarding

  • You should now be able to connect to the Portal Dedicated using https://localhost:8443

For users on a Windows machine running a browser to connect to the Portal Dedicated, run the Command Prompt as Administrator and type the following

C:\Windows> netsh interface portproxy v4tov4 listenport=8443
listenaddress=Windows_IP_Address connectport=8443
connectaddress=console_machine_ip_address

You should now be able to connect to the Portal Dedicated using https://localhost:8443

Browser won’t load the Portal Dedicated​

The Portal requires the browser to download some large (over 1MB) JavaScript files. Some networks run Intrusion Prevention Systems (IPS) that can block the JavaScript files from transferring over the network, leaving the user at the loading screen with no further activity. To determine if the IPS is causing a problem, try opening the Portal directly on the server running it or create a SSH tunnel to the server with the HTTP/S port the Portal Dedicated is available on. To fix the IPS incorrectly blocking the JavaScript file, consult the IPS documentation.

When Elasticsearch is run in a Linux machine, you might experience the following error:

Exception in thread "main"
org.elasticsearch.bootstrap.BootstrapException:
org.apache.lucene.index.CorruptIndexException: misplaced codec footer
(...)

To prevent this, increase the file descriptors limit running the following command as root:

$ ulimit -n 65536

Switching off Elasticsearch read-only mode​

To prevent an Elasticsearch node from running out of disk space Elasticsearch automatically switches the node to read-only mode when the disk usage reaches 95%.

When in this mode no new events will be indexed and Elasticsearch logs will contain related warnings:

[WARN ][o.e.c.r.a.DiskThresholdMonitor] [node] flood stage disk watermark [95%] exceeded on [CDKDg8hBTI-fMy6kE2FmhA][node][/var/lib/elasticsearch/nodes/0] free: 64.3mb[0.5%], all indices on this node will be marked read-only

For Elasticsearch 7.4.0 or later:

When disk usage on the affected node drops below the high watermark of 95%, Elasticsearch automatically removes the write block.

For Elasticsearch versions earlier than 7.4.0 the write block must be manually cleared.

The administrator must manually free up disk space on the server before proceeding.

To manually clear the write block and allow new events to be indexed:

curl -XPUT 'http://localhost:9200/_all/_settings' -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'

Elasticsearch does not need to be restarted after this request.

Connectivity to Elasticsearch failed when onboarding an agent with an early Java version​

Symptoms:

  • The agent fails to onboard.

  • You are using an early version of Java e.g. JDK 5u85 Hotspot.

  • This error is in the agent logs/console:

    Waratek error: waratek.controller.ControllerException: Connectivity test to Elasticsearch failed. Received fatal alert: protocol_version

  • This error is in the Elasticsearch logs:

    SSLHandshakeException: Client requested protocol TLSv1 is not enabled or supported in server context

Solution:

  1. Open your java security file:
sudo nano /usr/share/elasticsearch/jdk/conf/security/java.security

2. Find the jdk.tls.disabledAlgorithms list. It will look something like this:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4....

3. Remove "TLSv1" from the "jdk.tls.disabledAlgorithms" list, so that the list looks like this:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1.1, RC4....

4. Restart Elasticsearch

Handshake with Elasticsearch failed when onboarding an agent with Java 5/6​

Symptoms:

  • The agent fails to onboard.

  • You are using Java 5/6, lower than u91 definitely, e.g. jdk-jr-6u37.

  • This error is in the agent logs/console:

    Waratek error: Agent is unable to contact Management Console. This might be caused by an old Java version lacking required encryption algorithm.
    Waratek error: waratek.controller.ControllerException: Connectivity test to Elasticsearch failed. Received fatal alert: handshake_failure

  • This error is in the Elasticsearch logs:

    SSLHandshakeException: SSLv2Hello is not enabled

Solution:

  1. Open your java security file:
sudo nano /usr/share/elasticsearch/jdk/conf/security/java.security

2. Find the "jdk.tls.disabledAlgorithms" list. It will look something like this:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4....

3. Remove "TLSv1" from the "jdk.tls.disabledAlgorithms" list, so that the list looks like this:

jdk.tls.disabledAlgorithms=SSLv3, TLSv1.1, RC4....

4. Open your readonlyrest.yml file

sudo nano /etc/elasticsearch/readonlyrest.yml

5. Add “SSLv2Hello" to the “allowed_protocols” list, so that the list looks like this:

allowed_protocols: [SSLv2Hello,TLSv1,TLSv1.1,TLSv1.2]

6. Restart Elasticsearch

Elasticsearch fails to Start, with “node lock”​

Symptoms:

  • Elasticsearch fails to start and you see the following error in the elasticsearch log file:

    ... Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock

Solution:

  • Run the following command to fix the problem, by restoring required file permissions to the elasticsearch user:

    sudo chown -R elasticsearch: /var/lib/elasticsearch/*

The Above Solutions Didn’t Work​

Contact the appropriate system administrator or equivalent resources for your organisation to allow access from your machine(s).