Dropwizard 1.1 and Let's Encrypt with no Downtime

Published on: November 9, 2016

As of writing this, the free no gimmicks SSL certificate service, Let’s Encrypt, has issued nearly 15 million certificates. I’ve written about Let’s Encrypt before – it’s taking over the world and that’s a good thing.

This time I want to talk about Let’s Encrypt in the context of a web framework that I’ve gotten quite familiar with over the years, Dropwizard (I’m not going to link to all the articles I’ve written about Dropwizard because there’s so many!). Up until recently, Dropwizard needed to be restarted to apply a newly issued certificate, which doesn’t sound too bad, but a big plus with Let’s Encrypt is certificate renewel is frequent and automated. Some people like to refresh their certificates every month (instead of the required 90 days) and having guaranteed downtime every month is not a thought I relish. So I did what I had to do: submit a pull request. It was accepted and will be released as part of Dropwizard 1.1 (which is not yet released yet).

I do, very quickly, want to mention that this may not affect many people. It’s common to deploy Dropwizard behind a TLS termination proxy (HAProxy, apache, nginx), so you should refer to one of those guides when integrating Let’s Encrypt.

In this article, I want to provide a start to finish approach to setting a box with a standalone Dropwizard application.

The Code

First we’re going to start by creating a blank Dropwizard project through Maven.

mvn archetype:generate -DarchetypeGroupId=io.dropwizard.archetypes \
    -DarchetypeArtifactId=java-simple \
    -DarchetypeVersion=1.0.2

Since Dropwizard 1.1 is not released yet, an additional step is to clone the Dropwizard repo and mvn install. Then swap out the pom dependency version for 1.1.0-SNAPSHOT

We’ll add an endpoint that returns “Hello world”

@Path("/")
@Produces(MediaType.TEXT_PLAIN)
public class ExampleResource {
    @GET
    public String hello() {
        return "Hello world";
    }
}

There shouldn’t be anything too new at this point. What is new is registering the SslReloadBundle in our Application.

public class LetsDropwizardApplication extends Application<LetsDropwizardConfiguration> {

    public static void main(final String[] args) throws Exception {
        new LetsDropwizardApplication().run(args);
    }

    @Override
    public String getName() {
        return "LetsDropwizard";
    }

    @Override
    public void initialize(final Bootstrap<LetsDropwizardConfiguration> bootstrap) {
        bootstrap.addBundle(new SslReloadBundle());
    }

    @Override
    public void run(final LetsDropwizardConfiguration configuration,
                    final Environment environment) {
        environment.jersey().register(new ExampleResource());
    }

}

The SslReloadBundle will register a reload-ssl endpoint on the admin servlet that will loop through all registered HTTPS endpoints and reload certificate information. The reload uses the same information as the configuration that Dropwizard was started with (same keystore path, same keystore password, etc). A nice feature is that if reload fails due to an incorrect password, your application will continue using the last known good certificate, but make sure you fix it right away else the app will be unable to restart.

As an aside, I’ll be including the dropwizard-http2 module for that sweet, sweet HTTP2 endpoint. It will complicate some bits later on with the class path, but I’ll walk through that section as well.

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-http2</artifactId>
</dependency>

The Deployment

Time for deployment. I’ll be using DigitalOcean to host and using their lowest tier machine because I’m cheap. I’ll be getting the following for 16.8¢ per day:

1 core
512MB
20GB SSD
Ubuntu 16.04 (but there are many other options as well)

Careful now, a gust of wind could knock this machine over.

When creating a droplet, DigitalOcean has the option to boot with ssh keys. I recommend using them – there’s even a decent guide!

After logging in we’ll download and install the latest java as well as the Unlimited Strength Jurisdiction Policy Files, which will allow greater than 128bit key cryptography. I’ve had clients (non-browsers) unable to connect to servers, due to them requiring 256bit encryption. Note that this will only affect non-HTTP2 connections, as will be discussed later.

apt-add-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer unzip

curl http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip -L \
    -H "Cookie: gpw_e24=xxx; oraclelicense=accept-securebackup-cookie;" -o /tmp/jce_policy-8.zip && \
  unzip -j -o /tmp/jce_policy-8.zip -d /usr/lib/jvm/java-8-oracle/jre/lib/security
  rm -rf /tmp/jce_policy-8.zip

# Deploy jar file to /opt/lets 
mkdir -p /opt/lets

Next we’re going to install the Let’s Encrypt client and use the embedded server to request certificate information. We’re going to pass in a command flag that specifies that we want the certificate negotation to occur over port 80 because when we will want to renew the certificate, port 443 (the other port that the embedded server can bind to and the default HTTPS port) will be in use by our application.

apt-get install letsencrypt 

letsencrypt certonly --standalone -d test.nbsoftsolutions.com \
	--email <email> --agree-tos --standalone-supported-challenges http-01

The downside is that we’re unable to redirect HTTP requests to HTTPS, but I am more than willing to make this compromise because I can’t think of a situation where a client would request a plain HTTP request from our service and expect to redirected (this isn’t apache or nginx here!)

The certificate information needs to be massaged into a native Java format. For that we’ll be using openssl and keytool app installed in the standard Java direct. I’m using a dummy password. The same password is necessary to be used throughout.

cd /etc/letsencrypt/live/test.nbsoftsolutions.com
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out \
  /opt/lets/pkcs.p12 -name cert -password pass:123buckleMyShoe
cd /opt/lets
keytool -deststorepass 123buckleMyShoe -importkeystore -destkeypass 123buckleMyShoe \
  -destkeystore keystore.jks -srckeystore /opt/lets/pkcs.p12 -srcstoretype \
  PKCS12 -srcstorepass 123buckleMyShoe -alias cert

The Configuration

Now that we have our keystore tidied up, what does our config look like?

server:
  applicationConnectors:
    - type: h2
      port: 443
      keyStorePath: keystore.jks
      keyStorePassword: 123buckleMyShoe
      validateCerts: false
      validatePeers: false
  adminConnectors:
    - type: http
      port: 8081
      bindHost: 127.0.0.1

Couple things to note about this configuration:

Type of application server is h2, an HTTPS2 connection. Same configuration could have used for a regular https type as well.
Don’t be scared that validateCerts and validatePeers are false. Unfortunately, the default is true and if true, certificate validation will fail unconditionally. For more information, see the following issue.
The administration port is over plain HTTP and is listening to only 127.0.0.1 connections. This means that only a request originating from the box can communicate with the administration port and subsequently reload certificate information.
- In addition to binding to loopback, it also prudent to have properly configured firewall rules, so that no one can connect from the outside. See: how secure is binding to localhost in order to prevent remote connections
- The connection is plain http because we’ll be communicating with it via curl, and curl doesn’t support HTTP2 connections out of the box. One needs to compile and build nghttp2 and curl. I sleep better at night by knowing no sensitive information going to the admin port (this is normally a bad excuse to not implement HTTPS). This is just for demonstration purposes (could have used the https type as well).
- Cloudflare goes very deep into compiling curl for HTTP 2 in their article. May be a bit long.

The Operations

To run our application:

java -Xbootclasspath/p:alpn-boot-8.1.10.v20161026.jar \
  -jar lets.dropwizard-1.0-SNAPSHOT.jar server config.yaml

As promised earlier, we have to modify the boot classpath to include this jar. The gist is that Java8 doesn’t contain the necessary bits for Application-Layer Protocol Negotiation (ALPN), something required for HTTP 2. And since the required jar version changes for each JDK version see the Jetty guide for what version you need and overall usage.

I retrieved the version I needed by downloading straight from Maven Central

The Renew

Even more important to getting our first Let’s Encrypt certificate is keeping it renewed! For our purposes we’ll attempt to renew certificates at 2:30am every monday using cron:

30 2 * * 1 /opt/lets/refresh.sh

And the script itself:

#!/bin/bash

letsencrypt renew --agree-tos --standalone-supported-challenges http-01

cd /etc/letsencrypt/live/test.nbsoftsolutions.com
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem \
  -out /opt/lets/pkcs.p12 -name cert -password pass:123buckleMyShoe

cd /opt/lets
keytool -deststorepass 123buckleMyShoe -importkeystore -destkeypass 123buckleMyShoe \
  -destkeystore keystore.jks.tmp -srckeystore /opt/lets/pkcs.p12 -srcstoretype PKCS12 \
  -srcstorepass 123buckleMyShoe -alias cert

# Overwrite keystores
mv keystore.jks.tmp keystore.jks

CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST 'http://127.0.0.1:8081/tasks/reload-ssl')

if [[ "${CODE}" -neq "200" ]]; then
  echo "On no did not renew cert!" | ssmtp <email> 
fi

Things to note:

While the renew command is new the conversion to the Java keystore is very familiar. The only difference is an avoidance to overriding the default keystore before the new keystore is created.
A curl command to our adminstration port on 127.0.0.1 to reload certificate information. The HTTP status code is captured.
If the status code is not a 200 then we send an email so that I can fix the issue. For more information, see Send email alerts using ssmtp
If you want to force certificate renewal, pass --force to the command.

The Benchmark

For kicks and giggles I wanted to load test what our toy application can handle. I used h2load inside a docker container because the dependency list for was too long for convenience.

sudo docker run --rm -t svagi/h2load -n1000 -c100 -m10 https://test.nbsoftsolutions.com

finished in 7.16s, 139.72 req/s, 6.37KB/s
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 45.61KB (46707) total, 13.19KB (13507) headers (space savings 86.49%), 10.74KB (11000) data
                     min         max         mean         sd        +/- sd
time for request:   112.11ms       2.85s       1.22s    613.59ms    70.50%
time for connect:      2.76s       6.10s       3.94s       1.07s    73.00%
time to 1st byte:      2.95s       7.10s       5.06s       1.32s    53.00%
req/s           :       1.40        3.25        1.99        0.45    61.00%

140 requests per second is pretty measly for a Hello World application (see turning it up to eleven for performance tips). I saw the machine pegged at 100% CPU and memory usage. Hence this is why you shouldn’t skimp on resources.

The Details

Sslyze will let us know what cipher suites our server supports. When we just have HTTPS enabled (so no h2).

  * TLSV1_2 Cipher Suites:
      Preferred:
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384             ECDH-570 bits  256 bits      HTTP 200 OK
      Accepted:
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384             ECDH-570 bits  256 bits      HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384               DH-1024 bits   256 bits      HTTP 200 OK
        TLS_RSA_WITH_AES_256_CBC_SHA256                   -              256 bits      HTTP 200 OK
        TLS_RSA_WITH_AES_256_GCM_SHA384                   -              256 bits      HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256               DH-1024 bits   256 bits      HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384             ECDH-570 bits  256 bits      HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256               DH-1024 bits   128 bits      HTTP 200 OK
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256               DH-1024 bits   128 bits      HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256             ECDH-570 bits  128 bits      HTTP 200 OK
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256             ECDH-570 bits  128 bits      HTTP 200 OK
        TLS_RSA_WITH_AES_128_CBC_SHA256                   -              128 bits      HTTP 200 OK
        TLS_RSA_WITH_AES_128_GCM_SHA256                   -              128 bits      HTTP 200 OK

Notice the cipher suites with 256bit AES, and that is thanks to the unlimited strength crypto installed earlier.

Switching to h2, we see a slightly different story:

  * TLSV1_2 Cipher Suites:
      Preferred:
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256             ECDH-570 bits  128 bits      HTTP 200 OK
      Accepted:
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256             ECDH-570 bits  128 bits      HTTP 200 OK

Very interesting. Looks like the only cipher suite available is only one that is required from the HTTP 2 spec. I made sure to confirm this with the ssllabs tester. I’m not sure if this the desired default for Jetty.

Last but not least, this post doesn’t go into registering your dropwizard application as a service or starting it on boot (but it’s as easy as adapting an init script template and chkconfig on, respectively)

Comments

If you'd like to leave a comment, please email [email protected]