Spring Boot Applications in Kubernetes

Goals

Spring Boot is a very convenient framework for developing Java applications. It supports lots of useful features just by mentioning the package in your Gradle or Maven files. 

Building the application into a container and running the image in Docker allows you to have a very consistent experience in development, testing, and live. It also solves a lot of configuration and networking issues that used to be solved with a lot of custom scripting that usually broke *during* a maintenance window.

Running a distributed system (like a game server) requires deployment, and monitoring (called orchestration). The best solution for that right now is Kubernetes (K8S). It makes use of Containers to run the app, and watches its health. It can restart failed instances, and can dynamically and automatically scale up new instances based on load. When integrated with a cloud provider like Google or AWS, it can also spin up new servers (and you get charged a bit more, or a bit less when they scale down again). Another awesome benefit of K8S is that you can redeploy your whole server without a maintenance window. It will shut down an old Pod, and spin up a replacement with your new build. There are challenges to doing that in some cases that I'll talk through in a later post.

But there are a few tricks I've had to discover to make all this work well together:

• Have the docker Command be able to use variables so you can control things like startup memory usage.
• Have the JVM catch a signal and pass it to the application so it can shut down cleanly when K8S decides to scale down or replace a Pod. K8S goes through a two step process: it sends SIGINT first, then waits, then sends SIGKILL if the app doesn't shut down on its own.
• Get Spring Boot configuration into the application, and be able to override it with cluster-specific settings from a K8S ConfigMap.

Config Printing

The first challenge is *knowing* what configuration your app started with. There is no turn-key solution to this. So I include a little code that queries Spring and logs the config that was used to start the app. Very useful for debugging deployments. I also expose it to JMX (not shown):

    @Autowired
    ConfigurableEnvironment env;
    ... 

        MutablePropertySources sources = env.getPropertySources();

        // Find the name of every property...
        Set<String> uniqueNames = new HashSet<>();
        for (PropertySource<?> source : sources){
            if (source instanceof EnumerablePropertySource){
                uniqueNames.addAll(Arrays.asList(((EnumerablePropertySource) source).getPropertyNames()));
            }
        }

        // Use a TreeMap so the output is sorted.
        TreeMap<String, String> sortedProps = new TreeMap();

        for (String name : uniqueNames){
            // Read the property value, using Spring's resolution rules.
            String value = env.getProperty(name);
            if (name.toLowerCase().contains("pass") || name.toLowerCase().contains("secret")){
                value = "****"; // Don't show passwords.
            }
            sortedProps.put(name, value);
        }

        StringBuilder sb = new StringBuilder();
        sb.append("# Merged application properties.\n");

        // TODO: this doesn't really preserve the "file" exactly. E.g. special characters, continuations, comments, ...
        for (Map.Entry<String, String> el : sortedProps.entrySet()){
            sb.append(el.getKey()).append("\t=").append(el.getValue()).append("\n");
        }

        // String res = "EnvVars: " + envVars.toString() + " SystemProperties: " + systemProperties.toString() + " AppProperties: " + os.toString();
        String res = sb.toString();

This grabs all  the configuration names, then uses Spring's configuration system to look up the current value of each (based on its External Config priorities). It also hides passwords, because you shouldn't log such things.

Config Override

This is an example ConfigMap definition:

# kubectl apply -f server-config.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: server-config

data:
  server.port: "8090"
  api.server.port: "8090"
  management.server.port: "8090"

  logging.file: protoserver.log
  logging.level: INFO
  logging.level.com.protag.protoserver: INFO

  # The environment variable in the dockerfile that controls -Xmx${HEAP_SIZE} when the jvm starts in the container.
  HEAP_SIZE: 2400m

Deployment

This is an example deployment that picks up that configMap:

# kubectl apply -f server-deployment
# With "parameter" substitution:
# sed -s 's/BUILD_TAG/99/g' < server-deployment.yml | kubectl apply -f -
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: server
spec:
  replicas: 2
  template:
    metadata:
      labels:
        configmap-version: v8 # Fake label to force rolling update upon configMap change. Could use its md5 hash too.    
    spec:
      containers:
        - name: server
          image: localhost:5000/ourorg/server:BUILD_TAG
          # Pull config data in as environment variables, where springboot will merge them w application.properties.
          # There is currently no way to auto-redeploy the pod when this config changes. Some people hash the config.yml
          # file, and sed the hash into the deployment.yml, which is recognized as a big enough change to trigger
          # a RollingUpdate. Sigh.
          envFrom:
          - configMapRef:
            name: server-config
          # Make sure to only use "bash" where needed. "sh" strips dotted env vars, which spoils application.properties coming
          # through configMaps, and "env:" settings.
          ports:
          # the service
          - containerPort: 8090
            name: web
          - containerPort: 9010
            name: jmx

This uses envFrom to pull the ConfigMap above into the environment variables of the container. When Spring Boot starts up, those override what is in application.properties.

One odd thing, here is that "sh" will discard variable names that contain a dot. So use "bash", as you will see in the Dockerfile below.

The label configmap-version is used to force a rolling update if I change the configuration. I update that label and update the deployment. K8S will then restart each pod so it picks up that new config.

The string BUILD_TAG is used to ensure that an image produced by Jenkins or whatever is the actual one that K8S pulls from. Using LATEST is not reliable, and you can visibly see which image was used when running "kubectl describe".

Dockerfile

This is an example Dockerfile, used to create an image containing your Spring Boot app:

# Creates ourorg/server
FROM openjdk:11-jre-stretch
# May need a new base image. I don't see it on dockerhub anymore (only a few weeks later)

# Pick up variable passed in from gradle.build file
ARG JAR_FILE
ENV JAR_FILE=${JAR_FILE}

# JVM memory. Default is 128m. Make this less than the resource request in statefulSet.yml
ARG HEAP_SIZE=900m
ENV HEAP_SIZE=${HEAP_SIZE}

# We want the application.properties file to be loose, so it can be seen and edited more easily.
COPY ${JAR_FILE} application.properties /app/

# Run in /app to pick up application.properties, and drop logs there.
WORKDIR /app

# App and jmx ports
EXPOSE 8090/tcp 9010/tcp

# Allow SIGINT signals hit java app directly (by exec'ing over the initial shell). But still use a shell to
# expand variables. SIGINT is used by k8s to trigger graceful shutdown.
# Don't use "sh", it will strip dotted env vars, and you'll lose the configMap settings to override application.properties
ENTRYPOINT [ "bash", "-c", \
    "exec java -Xmx${HEAP_SIZE} \
               -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false \
               -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
               -jar ${JAR_FILE} \
          && echo Completed unexpected. >  /dev/termination-log"]

This Dockerfile allows the build system to specify the jar name, and starting java heap size. The way we are using these variables allows them to be overridden by K8S directly, or in a ConfigMap. Using variables in an ENTRYPOINT is not normally possible in Docker because it doesn't use a shell. Here, however, we explicitly run the command using "bash" as the shell. This allows the dotted environment variables to be passed through.

The reason for using "exec" in the ENTRYPOINT is so that the jvm is the process that receives the INT and KILL signals from K8S. Without that, the wrapping shell would not pass the INT signal to our application. It is needed to tell Spring Boot to start shutting down gracefully.

Summary

Well, I think that covers all my tricks for getting a Spring Boot app to work properly in Docker and Kubernetes. Let me know if you have questions, or trouble getting this working as I suggest. Or if there are other difficulties you've hit that you'd like help with.