Overview

There are a number of ways to manage the required certificates in Gloo Mesh. This overview will explain the different certificate and what they are used for.

Gloo Mesh Agents

Server/Client mTLS certificates are required for communication between the management plane agent (enterprise-networking) and the control plane agents (enterprise-agent) that are deployed to each cluster.

These agent certificates need to share the same root of trust but it is not required to have to be a part of the same root of trust as your Istio deployments.

If you are interested in generating your own certificates see Manually Provisioning Relay Certificates

  1. enterprise-networking (server certificate)

Example enterprise-networking Server Certificate Configuration

# enterprise-networking-server.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh
  1. enterprise-agent (client certificate) per cluster

Example enterprise-agent Signing CA configuration

# enterprise-agent-signing-ca.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = critical,CA:TRUE
keyUsage = digitalSignature, keyEncipherment, keyCertSign
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh

Example enterprise-agent Client Certificate configuration

NOTE The DNS name in the agent certificate must match the cluster name in the Gloo Mesh Management Plane or the agent will be rejected.

CLUSTER_NAME=cluster-1

# enterprise-agent-client.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = ${CLUSTER_NAME}"

Istio Deployments

Each Istio deployment requires that a CA certificate exists at the kubernetes secret cacerts in the istio-system namespace. Istio will use this certificate to issue workload certificates to each pod within the mesh.

If you would like to generate your own Istio CA certificates you can reference Manually Provisioning Istio CA Certs

  1. Istio CA Certificate per cluster

Example Istio Signing CA configuration

CLUSTER_NAME=cluster-1

# istio-signing-ca.conf
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
subjectAltName=@san
[ san ]
DNS.1 = istiod.istio-system.svc
[ req_dn ]
O = Istio
CN = Intermediate CA
L = ${CLUSTER_NAME}

Gloo Mesh Managed Approach

Gloo Mesh has a number of useful tools to help manage certificates and will be covered below. You must take care to make sure the solutions fit within your security guidelines.

Gloo Mesh Autogenerated Root CA for Agents (DEMO ONLY)**

By default the gloo mesh helm chart autogenerates its own Root CA certificate and Intermediate Signing CA for issuing server and client certificates to each enterprise-networking and enterprise-agent application. These secrets are stored in the gloo-mesh namespace under relay-root-tls-secret and relay-tls-signing-secret. The enterprise-agent certificate is stored in the gloo-mesh namespace on the client cluster under the name relay-client-tls-secret

Relay Certificates

When a new enterprise-agent application joins the gloo mesh management plane, it does so using simple TLS and a token that has been autogenerated within the helm chart. That token is trusted and the enterprise-networking application issues a client certificate to the agent. The agent then drops its connection and reestablishes an mTLS connection with the Management Plane.

Relay Exchange

This is a very handy feature when trying out Gloo Mesh features but should not be used in production.

This should be avoided in production if possible. The Root CA Certificate and key (unencrypted) are stored in kubernetes secrets. If for some reason they were to get deleted you would need to regenerate new certificates for all relay applications.

Gloo Mesh Autogenerated Root CA for Istio (DEMO ONLY)

Gloo mesh has the ability (via VirtualMesh) to generate and self-sign a Root CA certificate using the enterprise-networking application. The application will generate and self-sign a Root CA certificate that will be able to sign Istio Intermediate CA certificates when needed. Gloo Mesh will then place that Intermediate CA certificate in the client cluster under the kubernetes secret cacerts in the istio-system namespace.

Using VirtualMesh to autogenerate the Root CA and issue Intermediate CA certificates to each Istio deployment.

apiVersion: networking.mesh.gloo.solo.io/v1
kind: VirtualMesh
metadata:
  name: virtual-mesh
  namespace: gloo-mesh
spec:
  # specify gloo mesh certificate policy 
  mtlsConfig:
    # once a new intermediate CA certificate has been given to istio,
    # a restart is required of all pods in the mesh to receive new certificates.
    # Note: Do NOT use this autoRestartPods setting in production!!
    autoRestartPods: true
    shared:
      # root ca config
      rootCertificateAuthority:
        # autogenerate root CA certificate
        generated: {}
  federation:
    selectors:
    - {}
  meshes:
  - name: istiod-istio-system-cluster-1 
    namespace: gloo-mesh
  - name: istiod-istio-system-cluster-2
    namespace: gloo-mesh

Istio CA Cert

This is a very handy feature when trying out Gloo Mesh features but should not be used in production.

This should be avoided in production if possible. The Root CA Certificate and key (unencrypted) are stored in kubernetes secrets. If for some reason they were to get deleted you would need to regenerate new certificates gloo mesh and for every Istio deployment.

Saving the Autogenerate CA Certificates

It is important that if you rely on the autogenerated root certificates, you should download them and store them in a safe location in case they were to ever be lost by the management plane.

# Download Relay Root Certificate
mkdir relay-root-tls-secret
kubectl get secret -n gloo-mesh -o jsonpath='{.data.ca\.crt}' relay-root-tls-secret | base64 --decode > relay-root-tls-secret/ca.crt
kubectl get secret -n gloo-mesh -o jsonpath='{.data.tls\.key}' relay-root-tls-secret | base64 --decode > relay-root-tls-secret/tls.key

# Download Relay Signing Certificate
mkdir relay-tls-signing-secret
kubectl get secret -n gloo-mesh -o jsonpath='{.data.ca\.crt}' relay-tls-signing-secret | base64 --decode > relay-tls-signing-secret/ca.crt
kubectl get secret -n gloo-mesh -o jsonpath='{.data.tls\.key}' relay-tls-signing-secret | base64 --decode > relay-tls-signing-secret/tls.key

# For each VirtualMesh
# Download Root Certificate
VIRTUAL_MESH_NAME=gloo-mesh
SECRET_NAME=virtual-mesh.$VIRTUAL_MESH_NAME
mkdir $SECRET_NAME

kubectl get secret -n gloo-mesh -o jsonpath='{.data.root-cert\.pem}' $SECRET_NAME | base64 --decode > $SECRET_NAME/ca.crt
kubectl get secret -n gloo-mesh -o jsonpath='{.data.key\.pem}' $SECRET_NAME | base64 --decode > $SECRET_NAME/tls.key

Leveraging AWS ACM

There are a number of ways to use AWS ACM for your Root and Intermediate Certificate management strategy. Click the below links to see how

Using your own Certificates

For Relay

For relay communication you will need to issue a mTLS Server Certificate for the enterprise-networking application in the Management Plane and separate Client Certificates for each enterprise-agent application. These certificates need to share the same certificate chain, but can be different from the Istio CA Certificates chain.

See Certificates outside of Gloo Mesh

For Istio CAs

# Example kubectl apply (file names are important)
# ca-cert.pem
# ca-key.pem
# root-cert.pem
# cert-chain.pem
kubectl create secret generic cacerts -n istio-system \
      --from-file=ca-cert.pem=istio-ca/cert.pem \
      --from-file=ca-key.pem=istio-ca/ca.key.decrypted \
      --from-file=root-cert.pem=root-ca/ca.crt \
      --from-file=cert-chain.pem=istio-ca/cert-chain.pem

See Certificates outside of Gloo Mesh

Istio Plugin CA Certificate

Vault

Vault is an excelent tool to manage your Root and Intermediate Certificates. A writeup on how you can leverage vault within Gloo Mesh click below.

See Using Vault with Gloo mesh