Certificate architectures

Learn about how you can set up a certificate architecture for your Gloo Mesh Enterprise environment to maintain secure communication across your service meshes.

Certificate setup architectures

You can bring your own root CA certificate and intermediate signing CAs to sign the relay and Istio certificates that Gloo Mesh Enterprise requires for security.

After reviewing the example architectures, try out the guides for steps on how you can generate relay and Istio certificates for various certificate managers.

• AWS Certificate Manager (ACM)
• HashiCorp Vault
• Manually using your own tool, such as OpenSSL for self-signed demos or POCs

Number of root and intermediate CA certificates

The number of root and intermediate CA signing certificates increases both the complexity and security of your setup.

• Single root and intermediate
• 1 root, 2 intermediate
• Separate root and intermediate

If you favor simplicity over complexity, you can issue a single root CA and a single intermediate CA to issue certificates for both Istio and relay agents.

Another approach is to use one root CA but create two separate intermediate CAs for Istio and relay agents.

If security is your top concern, you can create separate root CAs for separate intermediate CAs. This setup ensures that the communication between the Gloo Mesh relay server and agents does not share a common root of trust with the communication of your service mesh applications.

Where the root and intermediate CA certificates are managed

Where you manage the root and intermediate CA certificates affects not only the security of your setup, but also the ability to use Gloo Mesh to automatically issue certificates.

• Both internal to cluster
• Hybrid approach
• Both external to cluster

You might store both the root and intermediate CA certificates within the management cluster, such as for demo setups. Gloo Mesh can manage both these CA certificates automatically, or you could integrate a certificate management tool like Vault in the cluster. To increase the security of this setup, you want to restrict access to the management cluster as much as possible.

You might opt store the root CA certificate outside the cluster, but leave the intermediate CA certificates for both relay and Istio in the management cluster. Or, you could store the root and relay intermediate CA certificates outside the cluster, but leave the Istio intermediate CA certificate in the management cluster. In either of these scenarios, Gloo Mesh continues to automatically issue Istio CA intermediate certificates from the management cluster. With this hybrid approach, you can more easily rotate the issuing CA certificate that Gloo Mesh uses at a later time.

If security is your top concern, you can store all of the root and intermediate CA certificates outside the cluster in a certificate management tool such as AWS Certificate Manager (ACM). While this approach assures that the signing certificates cannot be inadvertently accessed by resources within your service mesh setup, you are responsible for making sure that the certificates are managed and rotated properly. Gloo Mesh does not automatically issue the certificates.

Default Gloo Mesh-managed certificates

If you install Gloo Mesh Enterprise for exploration, testing, or proof-of-concept purposes, you can use the default root CA certificate and intermediate signing CAs that are automatically generated and self-signed by Gloo Mesh to secure communication between the management and data planes. Certificates are used by relay agents in workload clusters to secure communication with the relay server, and by Istio deployments to assign certificates to workload pods in the service mesh.

Federated trust process overview

You can set up Gloo Mesh Enterprise to use your root CA to sign requests from the intermediate CA that the Istio service mesh in each workload cluster uses. By setting up a shared root of trust for each mesh, you can federate and control how the destinations across meshes can securely communicate with each other. For more information, see the examples in Understand federated trust

1. First, the Gloo Mesh management server either creates the root CA or uses the custom root CA that you provided in the management cluster in the RootTrustPolicy custom resource.
2. Then, the Gloo Mesh agent on each of the mesh clusters creates a new private key and issued certificate pair. This key/certificate pair forms the intermediate CA that the mesh uses for the cluster. The private key does not leave the node.
3. Next, Gloo Mesh creates an internal API, the CertificateRequest custom resource, to request signed certificates from the certificate issuer in the management cluster's server.
4. Gloo Mesh signs certificate requests with the root CA on the management cluster and returns the signed certificate as a Kubernetes secret in the Mesh workload cluster.
5. The Istio service mesh in the workload cluster starts to use the new intermediate CA for its workloads. Note that you might have to reload the workloads to pick up the initial certificate change if you did not enable automatic pod bouncing.

Autogenerated relay root and intermediate CA certificates

To secure communication between the management and data planes, relay agents (gloo-mesh-agent) in workload clusters use server/client mTLS certificates to secure communication with the relay server (gloo-mesh-mgmt-server) in the management cluster.

By default, Gloo Mesh Enterprise autogenerates its own root CA certificate and intermediate signing CA for issuing the server and client certificates.

• The relay-server-tls-secret secret for the root CA certificate is stored in the gloo-mesh namespace of the management cluster.
• The root-trust.gloo-mesh secret for the relay server certificate is stored in the gloo-mesh namespace of the management cluster.
• The relay-client-tls-secret secret for the relay agent certificate is stored in the gloo-mesh namespace of each workload cluster.

When a workload cluster is registered with Gloo Mesh Enterprise, the initial setup of a secure communication channel between the management server and remote server follows this flow:

1. To validate authenticity, the relay agent uses simple TLS to transmit a token value, which is defined in relay-identity-token-secret on the workload cluster, to the relay server.
2. The token must match the value stored in relay-identity-token-secret on the management cluster, which is created during deployment of the relay server.
3. When the token is validated, the relay server generates a TLS client certificate for the relay agent.
4. The relay agent saves the client certificate in the relay-client-tls-secret.
5. All future communication from relay agents to the server, which uses the gRPC protocol, is secured by using mTLS provided by this certificate.

Autogenerated root CA certificate for Istio

The Istio deployment in each workload cluster requires a certificate authority (CA) certificate in the cacerts Kubernetes secret in the istio-system namespace.

Gloo Mesh Enterprise uses a root trust policy to configure the relay server (gloo-mesh-mgmt-server) to generate and self-sign a root CA certificate. This root CA certificate can sign Istio intermediate CA certificates whenever an Istio deployment in a workload cluster must create a certificate for a workload pod in its service mesh. Gloo Mesh stores the signed intermediate CA certificate in the cacerts Kubernetes secret in the istio-system namespace of the workload cluster.

Example root trust policy resource to autogenerate a root CA to sign intermediate CA certificates for each Istio deployment:

apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    autoRestartPods: true
    intermediateCertOptions: {}
    mgmtServerCa:
      generated:
        orgName: root-certificate-a

Saving the autogenerated certificates

The root CA certificates and unencrypted keys that Gloo Mesh Enterprise autogenerates are stored in Kubernetes secrets. If the secrets are deleted, you must regenerate new certificates for all relay agents and Istio deployments. Be sure to download them and store them in a safe location in the event of an accidental deletion.

1. From the management cluster, download the root certificate from the relay-server-tls-secret secret.

   mkdir relay-server-tls-secret
   kubectl get secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.ca\.crt}' relay-server-tls-secret | base64 --decode > relay-server-tls-secret/ca.crt
   kubectl get secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.tls\.key}' relay-server-tls-secret | base64 --decode > relay-server-tls-secret/tls.key
   
   

2. Download the relay signing certificate from the relay-tls-signing-secret secret.

   mkdir relay-tls-signing-secret
   kubectl get secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.ca\.crt}' relay-tls-signing-secret | base64 --decode > relay-tls-signing-secret/ca.crt
   kubectl get secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.tls\.key}' relay-tls-signing-secret | base64 --decode > relay-tls-signing-secret/tls.key
   

3. Download the Istio root signing certificate from each RootTrustPolicy resource.

   ROOT_TRUST_POLICY=gloo-mesh
   SECRET_NAME=root-trust.$ROOT_TRUST_POLICY
   mkdir $SECRET_NAME
   
   kubectl get secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.root-cert\.pem}' $SECRET_NAME | base64 --decode > $SECRET_NAME/ca.crt
   kubectl get secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.key\.pem}' $SECRET_NAME | base64 --decode > $SECRET_NAME/tls.key
   

4. In the event that the secrets are deleted, you can use these downloaded ca.crt and tls.key pairs to recreate the secrets in the management cluster. For example, if the relay-server-tls-secret is deleted, you can recreate the secret by running the following:

   kubectl -n gloo-mesh --context $MGMT_CONTEXT create secret tls relay-server-tls-secret \
     --cert=relay-server-tls-secret/ca.crt \
     --key=relay-server-tls-secret/tls.key
   

Insecure installations

If you prefer to set up Gloo Mesh without secure communication for quick demonstrations, you can install Gloo Mesh and register workload clusters in insecure mode. Insecure mode means that no certificates are generated to secure the connection between the Gloo Mesh server and agent components. Instead, the connection uses HTTP.

• In the meshctl install command to create the Gloo Mesh management plane, include the --set global.insecure=true flag.
• In the meshctl cluster register command to register a workload cluster, include the --relay-server-insecure=true flag.

Required certificates

Certificates are used by Gloo Mesh relay agents in workload clusters to secure communication with the Gloo Mesh relay server, and by Istio deployments to assign certificates to workload pods in the service mesh. Review the following certificates for the Gloo Mesh relay and Istio agents. Later, you can follow example steps set up the certificates in your cluster.

Certificates for Gloo Mesh relay server and agents

To secure communication between the management and data planes, relay agents (gloo-mesh-agent) in workload clusters use server/client mTLS certificates to secure communication with the relay server (gloo-mesh-mgmt-server) in the management cluster. These certificates must share the same root of trust. The following configurations are examples of the required certificates for relay components.

# gloo-mesh-mgmt-server.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh

# gloo-mesh-agent-signing-ca.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = critical,CA:TRUE
keyUsage = digitalSignature, keyEncipherment, keyCertSign
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh

CLUSTER_NAME=cluster1
# gloo-mesh-agent-client.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = ${CLUSTER_NAME}" # The DNS name must match the cluster name as it is registered with Gloo Mesh.

Certificates for Istio deployments

The Istio deployment in each workload cluster requires a certificate authority (CA) certificate in the cacerts Kubernetes secret in the istio-system namespace. Istio uses this intermediate CA certificate to issue workload certificates to each pod within its service mesh.

cat > "root-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
[ req_dn ]
O = Istio
CN = Root CA
EOF

CLUSTER_NAME=cluster1
cat > "cluster-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
subjectAltName=@san
[ san ]
DNS.1 = istiod.istio-system.svc
[ req_dn ]
O = Istio
CN = Intermediate CA
L = ${CLUSTER_NAME}
EOF