Using cert-manager with Kuma for mTLS

When configuring mutual TLS (mTLS) on the open source Kuma service mesh, users have a couple of different options. They can use a “builtin” certificate authority (CA), in which Kuma itself will generate a CA certificate and key for use in creating service-specific mTLS certificates. Users also have the option of using a “provided” CA, in which they must supply a CA certificate and key for Kuma to use when creating service-specific mTLS certificates. Both of these options are described on this page in the Kuma documentation. In this post, I’d like to explore the use of cert-manager as a “provided” CA for mTLS on Kuma.

Currently, Kuma lacks direct integration with cert-manager, so the process is a bit more manual than I’d prefer. If direct cert-manager integration is something you’d find useful, please consider opening an issue to that effect on the Kuma GitHub repository.

Assuming you have cert-manager installed already, the process for using cert-manager as the CA for a “provided” CA mTLS backend looks like this:

1. Define the root CA in cert-manager.
2. Prepare the secrets for Kuma.
3. Configure the Kuma mesh object for mTLS.

I know these steps are really too high level to be useful on their own, so I’ll step through them in a bit more detail in the sections below.

Define the mTLS Root CA

If you’ve browsed the cert-manager documentation, you’ve probably seen that there are a variety of ways to set up an Issuer (a means of issuing Certificate objects managed by cert-manager). In this example, I’ll just use a SelfSigned issuer, although you could use a different method. (To be honest, though, if you were going to go down the CA route you might as well just use the CA certificate and key with Kuma directly.)

To create a SelfSigned issuer, this piece of YAML would work:

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

Then use that SelfSigned issuer to issue a root CA for Kuma to use:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kuma-mtls-root-ca
  namespace: kuma-system
spec:
  isCA: true
  commonName: kuma-mtls-root-ca
  secretName: kuma-mtls-root-ca
  duration: 43800h # 5 years
  renewBefore: 720h # 30d
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - digital signature
    - key encipherment
    - cert sign
  issuerRef:
    name: selfsigned-issuer # References self-signed ClusterIssuer
    kind: ClusterIssuer
    group: cert-manager.io

Once this is applied to the cluster where cert-manager is running (which doesn’t necessarily need to be the cluster where Kuma will be installed, as you’ll see in a moment), then cert-manager will create not only the Certificate resource but also a corresponding Secret that contains the certificate and the private key (the name of that Secret is controlled by the spec.secretName field above). You’ll use that secret for the next step, which is preparing the secrets for Kuma.

Preparing Secrets for Kuma

It would be nice if Kuma could just use/read the Secret created by cert-manager directly, but that functionality doesn’t currently exist. As a result, we’ll need to take the Kubernetes Secret created by cert-manager and turn it into something that Kuma can leverage.

To do this, you’ll need to make two mesh-scoped Kuma secrets (see the Kuma secrets documentation). On Kubernetes, these will be Kubernetes Secrets, but labeled for a specific mesh and tagged as a specific type.

This example template shows what a mesh-scoped Kuma secret would look like on Kubernetes:

---
apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
data:
  value: blah
type: system.kuma.io/secret

The label (kuma.io/mesh: default), the type (system.kuma.io/secret), and the namespace (kuma-system) mark this as a mesh-scoped Kuma secret. How do you take the Kubernetes Secret generated by cert-manager and turn it into one of these?

Well, you could export the contents of the Kubernetes Secret created by cert-manager (using kubectl get secret <name> -jsonpath='{.data.tls\.crt}', piping that through base64 to decode it, and then redirecting to a file), and then use the file to create a new mesh-scoped Kuma secret using the template above, making sure to re-encode the file contents with base64 alone the way. However, there’s no real need to have the CA certificate and key in a file on the disk, and there’s no real sense in decoding and then re-encoding the Secret’s contents. Is there a better way?

In fact, there is. Using yq (here’s the GitHub repository for it), you can do what you need with a single command (I’ll break this command down in a moment):

yq e ".data.value = \
\"$(kubectl -n kuma-system get secret kuma-mtls-root-ca \
-jsonpath='{.data.tls\.crt}')\" secret-template.yaml | \
yq e '.metadata.name = "mtls-root-certificate"' -

That’s a doozy! What’s happening here? You’re using yq to modify the values of a couple of fields; there’s two iterations of yq, each modifying a different field. The first iteration of yq modifies the .data.value field, making its value equal to the output of the kubectl get secret command (which is the base64-encoded CA certificate in the Secret created by cert-manager). yq expects double quotes around the value being assigned to the field, but double quotes are needed to do the command substitution so the “inner quotes” are backslash-escaped. The second iteration of yq modifies the .metadata.name field to be equal to “mtls-root-certificate”. Since there’s no command substitution in this iteration, you’re using single quotes around the yq operation and unescaped double quotes around the value.

The end result looks something like this:

---
apiVersion: v1
kind: Secret
metadata:
  name: mtls-root-certificate
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
data:
  value: LS0tLS1CRUdJTiBDRVJUSUZJ... # Base64-encoded contents
type: system.kuma.io/secret

Just add one final pipe (pipe it into kubectl apply -f -) to the command above and you’re good to go.

Repeat for the key:

yq e ".data.value = \
\"$(kubectl -n kuma-system get secret kuma-mtls-root-ca \
-jsonpath='{.data.tls\.key}')\" secret-template.yaml | \
yq e '.metadata.name = "mtls-root-key"' - | \
kubectl apply -f -

You’re now ready for the final step, which is configuring the Kuma mesh for mTLS.

Configuring the Mesh for mTLS

Fortunately, this step is pretty straightforward. Working from the example found in the Kuma mTLS documentation for a “provided” backend, something like this would work:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: cert-manager
    backends:
      - name: cert-manager
        type: provided
        dpCert:
          rotation:
            expiration: 1d
        conf:
          cert:
            secret: mtls-root-certificate
          key:
            secret: mtls-root-key

The backend name is entirely subjective, of course, but the names of the secrets referenced by spec.conf.cert.secret and spec.conf.key.secret must match the names of the mesh-scoped Kuma secrets you created in the previous section.

And that’s it! Once this configuration is in place, Kuma will use the CA certificate issued by cert-manager (which you then translated into a pair of mesh-scoped Kuma secrets) to automatically issue and sign the mTLS certificates used by the data plane proxies. (You could use the Envoy admin API to see more details on these certificates, but that’s a different post for a different day.)

If you have any questions about this, or about Kuma in general, feel free to hit me up. You can find me in the Kuma community Slack, or hit me on Twitter. My DMs are open and I’d be happy to help if I’m able.

Metadata and Navigation