Agent configuration/startup troubleshooting and debugging options

Applications running on IBM J9 JDK with Java Agent v25.0.0+ may throw java.lang.VerifyError at JVM start up. Use -Xverify:none Java Option

On Java Agent v25.0.0 and later, a java.lang.VerifyError: JVMVRFY012 stack shape inconsistent error may be thrown by certain IBM J9 JDKs at JVM start-up, whether the agent is connecting to the Portal or running standalone. Should this error occur, the following Java option should be added

-Xverify:none

Applications running on JDK 16+. On-boarding to Portal Dedicated may require additional start up Java options in some cases

On JDK 16+, the following additional Java option may need to be added at JVM start up in some cases.

--add-opens=java.management/sun.management=ALL-UNNAMED

On JDK 16+, applications may throw the following IllegalAccessError when on-boarding to Portal Dedicated with ES running on HTTPS

java.lang.IllegalAccessError: superclass access check failed: class waratek.connectivity.CustomTrustStoreSSLHttpConnection$CustomUrlConnectionHandler

In this case, the following Java option should be added at JVM start up.

--add-opens=java.base/sun.net.www.protocol.https=ALL-UNNAMED

On JDK 16+, applications may throw the following IllegalAccessError when on-boarding to Portal Dedicated with ES running on HTTP

java.lang.IllegalAccessError: class waratek.connectivity.http.UrlFactory$2 (in unnamed module) cannot access class sun.net.www.protocol.http.Handler (in module java.base) because module java.base does not export sun.net.www.protocol.http to unnamed module

In this case, the following Java options should be added at JVM start up.

--add-opens=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.http=ALL-UNNAMED

Java Security Manager policy causes application to throw AccessControlException when the Waratek Agent is added.

IBM WebSphere ND will fail to start with java.security.AccessControlException when the Waratek Agent is added, if the WebSphere JDK’s java.policy file is not updated as outlined in the “Java Agent Installation Guide - IBM WebSphere” section.

While WebSphere ND is the one application server type where this is known to occur, other applications could have Java Security Manager policy files which could cause the application to throw AccessControlException when the Waratek Agent is added;

java.security.AccessControlException: access denied ("oracle.security.jps.JpsPermission")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:886)

Should an AccessControlException occur, add the following entry into one of the active 'java.policy' files (making sure to replace <absolute-path-to-waratek-software> with the correct path);

grant codeBase "file:<absolute-path-to-waratek-software>/agent/*" {permission java.security.AllPermission;};

Option to disable Security manager.

There could be rare cases where a Java Security Manager policy in place can not be edited and may prevent the Waratek Agent from working as expected. On Java Agent v25.0.0+ , a new Waratek debugging option has been introduced which disables the Java Security Manager.

com.waratek.debug.DisableSecurityManager=true

Agent connection to Portal drops for Oracle WebLogic applications

Some application servers (e.g. Oracle WebLogic) configure UrlHandlerFactory to use non-default, custom handlers for HTTP/HTTPS protocol. This may cause connectivity issues for the Agent connecting to the Portal, and the Agent connection may drop. The application continues running, but the Agent status changes from Online to Offline in the Portal. If this issue occurs, it will occur within the first few minutes of the Agent coming Online. Should it occur, add the debug flags detailed in the Debug logging option for Agent-Portal communication issues section of this page, and restart the application server. A java.net.SocketException: Socket closed error should be logged, prior to the Agent going Offline. Add the following Waratek option to the startup command line, setting the value to true. Enabling this option forces the Agent to use "UseSunHttpHandler" http handlers. Restart the application server after setting the option.

com.waratek.UseSunNetHttp=true

This com.waratek.UseSunNetHttp Waratek option is available on Java Agent v25.0.0+ . The default value, when not set, is false.

Option com.waratek.UseJavaHttpProxy forces Agent Portal communication to use proxyHost and proxyPort properties set by the Java application.

The com.waratek.UseJavaHttpProxy Waratek option is introduced in Java Agent v25.0.0. In Agent releases prior to v25.0.0, Java applications which happened to set http.proxyHost, http.proxyPort ,https.proxyHost, orhttps.proxyPort properties on the Java command line (or as System Properties used by the Java application) encountered a communication issue connecting to the Portal. That issue has been addressed in Waratek Agent v25.0.0. If the proxyHost and/or proxyPort properties are set by the Java application, they do not now affect Agent-Portal communication and no extra Waratek options are required. In the unlikely (and non-recommended) case, that the user wants to force the Agent to use these properties for Agent Portal communication, then the following flag is available. By setting the value to true, this will force the Agent v25.0.0+ Portal communication behaviour to be as it was Agent versions prior to v25.0.0. Restart the application server after setting the option.

com.waratek.UseJavaHttpProxy=true

The default value, when not set, is false.

Debug logging option for Agent-Portal communication issues.

To enable verbose debug logging in cases where Agent-Portal communication issues occur, the following Waratek option may be added at JVM start up.

com.waratek.debug.mc=true

Setting the following Waratek option will redirect debug messages into the specified debug log file .

com.waratek.debug.transport.file=<debug log file>

Where the logs obtained from the debug flags above do not provide sufficient information as to the cause of the communication issues, a further debugging approach which may be useful is as follows.

Check connectivity from the Agent to Portal is ok by running the following curl command:

curl -v 'https://agent-api-2.waratek.com/api/2/agents' --data '{"runtimeLanguage": "JAVA"}'

to see the curl response contains an expected 403 response code;

HTTP/2 403

If the Curl connection works but the Agent doesn’t, it’s possible that the issue is with the version of Java. The sample PortalRestClient debugging application below attempts to establish a connection to the Portal, in a similar way to what the Java Agent does internally, except that it doesn’t ignore any exceptions so can be more useful when debugging. Compile PortalRestClient.java with the same version of Java that the Agent is using:

/path/to/java/bin/javac PortalRestClient.java

and run it:

/path/to/java/bin/java PortalRestClient

The expected output should include a 403 response:

start doConnection
Forcing TLSv1.2 was successful
response: 403
end doConnection

The PortalRestClient.java source code:

import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.security.SecureRandom;

import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;

public class PortalRestClient {
private static final String DEFAULT_HOST = "https://agent-api-2.waratek.com";
private static final String DEFAULT_ENDPOINT = "/api/2/agents/status";
private static final String PORTAL_STATUS_METHOD = "PUT";
private static final int DEFAULT_CONNECTION_TIMEOUT_MS = 20 * 1000; // 20 seconds
private static final int DEFAULT_READ_TIMEOUT_MS = 50 * 1000; // 50 seconds
private static final String TLS_V1_2 = "TLSv1.2";

    public static void main(String[] args) throws Exception {
        try {
            String host = DEFAULT_HOST;
            if (args != null && args.length > 0) {
                host = args[0];
            }
            doConnection(host + DEFAULT_ENDPOINT);
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    private static void doConnection(String url) throws Exception {
        System.out.println("start doConnection to " + url);
        HttpURLConnection conn = (HttpURLConnection) new URL(url).openConnection();
        customizeConnection(conn);
        forceTls12IfSupported(conn);
        conn.setDoInput(true);
        conn.setDoOutput(true);
        conn.setRequestMethod(PORTAL_STATUS_METHOD);

        OutputStream connStream = conn.getOutputStream();
        connStream.write("hello".getBytes());
        connStream.close();
        conn.connect();

        System.out.println("response: " + conn.getResponseCode());

        conn.disconnect();
        System.out.println("end doConnection");
    }

    /**
     * Common settings for any connection made by the Agent to Portal
     */
    private static void customizeConnection(HttpURLConnection conn) {
        conn.setUseCaches(false);
        conn.setInstanceFollowRedirects(false);
        conn.setConnectTimeout(DEFAULT_CONNECTION_TIMEOUT_MS);
        conn.setReadTimeout(DEFAULT_READ_TIMEOUT_MS);
    }

    private static void forceTls12IfSupported(HttpURLConnection conn) {
        if (!(conn instanceof HttpsURLConnection)) {
            return;
        }
        if (!isTls12Supported()) {
            System.out.println("Not trying to force " + TLS_V1_2);
            return;
        }
        if (doForceTls12IfSupported((HttpsURLConnection) conn)) {
            System.out.println("Forcing " + TLS_V1_2 + " was successful");
        } else {
            System.out.println("Forcing " + TLS_V1_2 + " has failed");
        }
    }

    public static boolean isTls12Supported() {
        try {
            SSLContext.getInstance(TLS_V1_2);
            return true;
        } catch (Exception ignore) {}

        return false;
    }

    private static boolean doForceTls12IfSupported(HttpsURLConnection secureConn) {
        try {
            SSLContext sslContext = SSLContext.getInstance(TLS_V1_2);
            sslContext.init(null, null, new SecureRandom());
            secureConn.setSSLSocketFactory(sslContext.getSocketFactory());
            return true;
        } catch (Exception tlsException) {
            tlsException.printStackTrace();
        }
        return false;
    }

}

Agent-Portal Dedicated communication issues on early Java 7 JDKs, when Elasticsearch is running over HTTP

It is recommended that Elasticsearch is configured to run over HTTPS. However, in cases where Elasticsearch is running over HTTP, and when the application running on early Java7 JDKs (7u121 or earlier), the Agent onboarding may fail with ControllerException ;

waratek.controller.ControllerException: Failure while trying to register an Agent: Received fatal alert: protocol_version

Adding the -Dhttps.protocols=TLSv1.2 Java Option to the application startup will allow the Agent to connect in this case.

Debug logging options for ARMR Patch Compilation, Execution, and Errors

Three debug logging Waratek options are introduced in Java Agent v25.0.0 for logging ARMR Patch compilation, execution and error details.

• com.waratek.ShowPatchCompilation=true
• com.waratek.ShowPatchExecution=true
• com.waratek.ShowPatchErrors=true

The default values for each option, if not specified, is false.

Debug logging will be written (with some other debug messages) to the file configured by the following Waratek option:

• com.waratek.debug.log=<path/to/waratek_debug.log>

Log entry prefixes:

• com.waratek.ShowPatchCompilation and com.waratek.ShowPatchExecution log entries are logged with the prefix [INFO.ARMR]
• com.waratek.ShowPatchErrors log entries are logged with the prefix [ERROR.ARMR]

HTTP/s requests to the protected application are failing with IllegalStateException: Oceanic Agent Servlet API is now configured for <'javax/servlet' | 'jakarta/servlet'> namespace

Applications using the jakarta.servlet namespace (e.g. applications running in Tomcat10+, JBoss EAP 8+ or any server which supports Jakarta Servlet 5.0+ specification) will require the following Waratek property to be set in the waratek.properties file

com.waratek.servlet=jakarta

If not set, then the following error will be thrown on when the servlet processes HTTP/s requests

java.lang.IllegalStateException: Oceanic Agent Servlet API is now configured for 'javax/servlet' namespace. Securing Application which utilizes 'jakarta.servlet.GenericFilter' is not possible.

Conversely, if the protected application is using the javax.servlet namespace but is configured with com.waratek.servlet=jakarta then the following error will be thrown

java.lang.IllegalStateException: Oceanic Agent Servlet API is now configured for 'jakarta/servlet' namespace. Securing Application which utilizes 'javax.servlet.http.HttpServlet' is not possible.

In this case, the com.waratek.servlet=jakarta property should be removed (so the default com.waratek.servlet value javax is used), or be explicitly set to com.waratek.servlet=javax .