Enabling NIO on a working SSL example causes newConnection to hang forever

[Yaytay]

The following unit test passes if the useNio() line is commented out, but hangs forver (well, I left it overnight) if NIO is used.
The test is just an attempt to perform a minimal SSL connection to a rabbitmq server, dynamically created using the testcontainers library.
The various certificate files were created according to the instructions in the documentation, but will be attached here.

It wouldn't be surprised to find that additional configuration is required for SSL with NIO, but in the absence of that config it should either "just work" or throw an exception.

The order of useNio and useSslProtocol doesn't affect the hang.

import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import java.io.InputStream;
import java.security.KeyStore;
import java.util.stream.Collectors;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.utility.MountableFile;

import static org.junit.Assert.assertNotNull;

/**
 *
 * @author jtalbut
 */
public class RabbitMQSslRawTest {
  
  @SuppressWarnings("constantname")
  private static final Logger logger = LoggerFactory.getLogger(RabbitMQSslRawTest.class);
  
  public static final GenericContainer rabbitmq = getRabbitMqContainer();  

  public static GenericContainer getRabbitMqContainer() {
    GenericContainer container = new GenericContainer("rabbitmq:3.9.8-management-alpine")
          .withCopyFileToContainer(MountableFile.forClasspathResource("/ssl-server/rabbitmq.conf"), "/etc/rabbitmq/rabbitmq.conf")
          .withCopyFileToContainer(MountableFile.forClasspathResource("/ssl-server/ca/ca_certificate.pem"), "/etc/rabbitmq/ca_certificate.pem")
          .withCopyFileToContainer(MountableFile.forClasspathResource("/ssl-server/server/server_certificate.pem"), "/etc/rabbitmq/server_certificate.pem")
          .withCopyFileToContainer(MountableFile.forClasspathResource("/ssl-server/server/private_key.pem"), "/etc/rabbitmq/server_key.pem")
          .withExposedPorts(5671, 5672, 15672)
                ;
    container.start();
    logger.info("Started test instance of RabbitMQ with ports {}"
            , container.getExposedPorts().stream().map(p -> Integer.toString((Integer) p) + ":" + Integer.toString(container.getMappedPort((Integer) p))).collect(Collectors.toList())
    );
    return container;
  }
  
  @Test
  public void testSsl() throws Throwable {
    
    char[] trustPassphrase = "password".toCharArray();
    KeyStore tks = KeyStore.getInstance("JKS");
    InputStream tustKeyStoreStream = this.getClass().getResourceAsStream("/ssl-server/localhost-test-rabbit-store");
    tks.load(tustKeyStoreStream, trustPassphrase);

    TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
    tmf.init(tks);

    SSLContext c = SSLContext.getInstance("TLSv1.3");
    c.init(null, tmf.getTrustManagers(), null);

    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    factory.setPort(rabbitmq.getMappedPort(5671));
    factory.useSslProtocol(c);
    factory.useNio();

    Connection conn = factory.newConnection();
    assertNotNull(conn);
    conn.close();
    logger.info("Connected");

  }

}

[Yaytay]

ssl-server.zip

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

We cannot suggest much without knowing what platform, JDK version, RabbitMQ version and Erlang version this was tested against, as well as the exact server configuration, perhaps with keys produced by tls-gen.

Server logs and a traffic capture would help immensely, too.

[Yaytay]

Platform: Windows 10.
JDK: Azul 17 (OpenJDK Runtime Environment Zulu17.28+13-CA (build 17+35-LTS))
RabbitMQ: rabbitmq:3.9.8-management-alpine
Erlang Version: 24.1.2 [jit]
TLS Library: OpenSSL - OpenSSL 1.1.1l 24 Aug 2021
Server configuration and keys are in the zip file attached to my first comment.

I'll get logs and traffic capture.

[Yaytay]

Docker output and packet capture.
RabbitNioSslCapture.zip
RabbitDockerOutput.txt

[Yaytay]

It also fails on a VM running Debian 10.11 with zulu17.30.15-ca-jdk17.0.1-linux_x64.

[acogoluegnes]

@Yaytay I guess this is the original post on the mailing list: https://groups.google.com/g/rabbitmq-users/c/LGlpHNx5QIQ/m/AobNKVegAAAJ.

I managed to reproduce. The TLS handshake fails with NIO and TLS 1.3. A workaround is to use blocking IO or TLS 1.2, until we come up with a fix.

[Yaytay]

All the things I tried playing around with and I never tried dropping to TLSv1.2!! :)
Thanks, I can confirm that that works.

And yes, that's me on the mailing list.

How can I track work on this, will the issue I raised be re-enabled?

[acogoluegnes]

Yes, we'll create a follow-up issue.

[Yaytay]

Thank you.