You then use these credentials to create an intermediate CA certificate and key that can be used by Gloo Mesh Enterprise to automatically sign and issue client TLS certificates for the workload clusters.

For more information about this approach, see Bring your own CAs with automatic client TLS certificate rotation.

Single cluster

Use the steps in this guide to generate custom TLS certificates with OpenSSL.

Create the root CA credentials

Create root CA credentials that are stored in the cluster. Note that for security purposes, you might store the root CA credentials separately from the cluster. Then, you use the root CA to sign a subsequent intermediate CA certificate and key that are stored in the cluster. These intermediate CA certificate and key become the “root” CA certificate and key in the subsequent steps.

  1. Create a self-signed root CA certificate and key.
      openssl req -new -newkey rsa:4096 -x509 -sha256 \
     -days 3650 -nodes -out relay-root-ca.crt -keyout relay-root-ca.key \
     -subj "/CN=relay-root-ca" \
     -addext "keyUsage = keyCertSign"
      
  2. If it doesn’t already exist, create the gloo-mesh namespace.
      kubectl create namespace gloo-mesh 
      
  3. Store the root CA certificate in the relay-root-tls-secret Kubernetes secret.
      kubectl create secret generic relay-root-tls-secret -n gloo-mesh \
     --from-file=tls.crt=relay-root-ca.crt \
     --from-file=ca.crt=relay-root-ca.crt \
     --from-file=tls.key=relay-root-ca.key 
      

Create the intermediate CA credentials

Use the root CA key to generate an intermediate CA certificate and key. These credentials are later used to sign client TLS certificates for the Gloo agent.

  1. Create the configuration for the intermediate CA.

      cat > "relay-intermediate-ca.conf" <<EOF
    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:TRUE
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid:always,issuer:always
    keyUsage = digitalSignature, keyEncipherment, keyCertSign
    extendedKeyUsage = clientAuth, serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS = *.gloo-mesh
    EOF
      
  2. Generate the private key for the intermediate CA.

      openssl genrsa -out "relay-int.key" 2048
      
  3. Generate the certificate signing request (CSR).

      openssl req -new -key "relay-int.key" -out relay-int.csr -subj "/CN=relay-int"
      
  4. Sign the CSR with the root CA key.

      openssl x509 -req \
     -days 3650 \
     -CA relay-root-ca.crt -CAkey relay-root-ca.key \
     -set_serial 0 \
     -in relay-int.csr -out relay-int.crt \
     -extensions v3_req -extfile "relay-intermediate-ca.conf"
      
  5. Save the intermediate CA certificate and key in the relay-tls-signing-secret Kubernetes secret.

      kubectl create secret generic relay-tls-signing-secret -n gloo-mesh \
     --from-file=tls.crt=relay-int.crt \
     --from-file=tls.key=relay-int.key \
     --from-file=ca.crt=relay-root-ca.crt 
      

Create the server TLS certificate

Use the root CA credentials to generate a TLS certificate for the Gloo management server.

  1. Create the configuration for the server TLS certificate.
      cat >"relay-server.conf" <<EOF
    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth, serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS = *.gloo-mesh
    EOF
      
  2. Generate the private key for the Gloo management server.
      openssl genrsa -out "relay-server.key" 2048
      
  3. Generate the certificate signing request for the Gloo management server.
      openssl req -new -key "relay-server.key" -out "relay-server.csr" -subj "/CN=*.gloo-mesh" -config "relay-server.conf"
      
  4. Use the root CA credentials that you created earlier to sign the certificate signing request and create the server TLS certificate.
      openssl x509 -req \
     -days 3650 \
     -CA relay-root-ca.crt -CAkey relay-root-ca.key \
     -set_serial 0 \
     -in relay-server.csr -out relay-server.crt \
     -extensions v3_req -extfile "relay-server.conf"
      
  5. Store the server TLS certificate, private key, and root CA in the relay-server-tls-secret Kubernetes secret.
      kubectl create secret generic relay-server-tls-secret -n gloo-mesh \
     --from-file=tls.crt=relay-server.crt \
     --from-file=tls.key=relay-server.key \
     --from-file=ca.crt=relay-root-ca.crt
      

Create the certificate chain for the telemetry collector

  1. Store the root CA certificate in the telemetry-root-secret Kubernetes secret.
      kubectl create secret generic telemetry-root-secret \
     --from-file=ca.crt=relay-root-ca.crt \
     --namespace gloo-mesh
      

Create relay identity tokens

To establish initial trust between the Gloo management server and agent, you must set up your own relay identity tokens.

  1. Create an environment variable with your identity token. The token can be any string value.

      export TOKEN="<identity_token>"
      
  2. Store the token in the relay-identity-token-secret Kubernetes secret.

      kubectl create secret generic relay-identity-token-secret -n gloo-mesh --from-literal=token=$TOKEN
      

Install Gloo Mesh Core

  1. Follow the Install with Helm guide.
  2. In your Helm values file, add the following values.
      glooMgmtServer: 
      enabled: true
      registerCluster: true
      serviceType: ClusterIP
      relay:
        disableCaCertGeneration: true
        signingTlsSecret:
          name: relay-tls-signing-secret
          namespace: gloo-mesh
        tlsSecret:
          name: relay-server-tls-secret
          namespace: gloo-mesh
        tokenSecret:
          key: token
          name: relay-identity-token-secret
          namespace: gloo-mesh
    glooAgent: 
      enabled: true
      relay:
        authority: gloo-mesh-mgmt-server.gloo-mesh
        serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900
        clientTlsSecretRotationGracePeriodRatio: ""
        rootTlsSecret:
          name: relay-root-tls-secret
          namespace: gloo-mesh
        tokenSecret:
          key: token
          name: relay-identity-token-secret
          namespace: gloo-mesh
    telemetryCollector: 
      enabled: true
      extraVolumes: 
        - name: root-ca
          secret:
            defaultMode: 420
            optional: true
            secretName: telemetry-root-secret
        - configMap:
            items:
              - key: relay
                path: relay.yaml
            name: gloo-telemetry-collector-config
          name: telemetry-configmap
        - hostPath:
            path: /var/run/cilium
            type: DirectoryOrCreate
          name: cilium-run
      
    Helm valueDescription
    glooMgmtServer.relay.disableCaCertGenerationDisable the generation of self-signed certificates to secure the relay connection between the Gloo management server and agent.
    glooMgmtServer.relay.signingTlsSecretAdd the name and namespace of the Kubernetes secret that holds the intermediate CA credentials that you created earlier.
    glooMgmtServer.relay.tlsSecretAdd the name and namespace of the Kubernetes secret that holds the server TLS certificate for the Gloo management server that you created earlier.
    glooMgmtServer.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you created earlier.
    glooAgent.relay.rootTlsSecretAdd the name and namespace of the Kubernetes secret that holds the root CA credentials that you created earlier.
    glooAgent.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you created earlier.
    telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

Multicluster

Use the steps in this guide to generate custom TLS certificates with OpenSSL.

Create the root CA credentials

Create root CA credentials that are stored in the management cluster. Note that for security purposes, you might store the root CA credentials separately from the cluster. Then, you use the root CA to sign a subsequent intermediate CA certificate and key that are stored in the cluster. These intermediate CA certificate and key become the “root” CA certificate and key in the subsequent steps.

  1. Create a self-signed root CA certificate and key.

      openssl req -new -newkey rsa:4096 -x509 -sha256 \
     -days 3650 -nodes -out relay-root-ca.crt -keyout relay-root-ca.key \
     -subj "/CN=relay-root-ca" \
     -addext "keyUsage = keyCertSign"
      
  2. If it doesn’t already exist, create the gloo-mesh namespace in each cluster.

      kubectl create namespace gloo-mesh --context $MGMT_CONTEXT
    kubectl create namespace gloo-mesh --context $REMOTE_CONTEXT1
    kubectl create namespace gloo-mesh --context $REMOTE_CONTEXT2
      
  3. Store the root CA certificate in the relay-root-tls-secret Kubernetes secret on the management cluster.

      kubectl create secret generic relay-root-tls-secret -n gloo-mesh \
     --from-file=tls.crt=relay-root-ca.crt \
     --from-file=ca.crt=relay-root-ca.crt \
     --from-file=tls.key=relay-root-ca.key \
     --context ${MGMT_CONTEXT} 
      
  4. Copy the root CA certificate to each workload cluster.

      kubectl create secret generic relay-root-tls-secret -n gloo-mesh --context $REMOTE_CONTEXT1 --from-file ca.crt=relay-root-ca.crt
    kubectl create secret generic relay-root-tls-secret -n gloo-mesh --context $REMOTE_CONTEXT2 --from-file ca.crt=relay-root-ca.crt
      

Create the intermediate CA credentials

Use the root CA key to generate an intermediate CA certificate and key. These credentials are later used to sign client TLS certificates for the Gloo agents on each workload cluster.

  1. Create the configuration for the intermediate CA.

      cat > "relay-intermediate-ca.conf" <<EOF
    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:TRUE
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid:always,issuer:always
    keyUsage = digitalSignature, keyEncipherment, keyCertSign
    extendedKeyUsage = clientAuth, serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS = *.gloo-mesh
    EOF
      
  2. Generate the private key for the intermediate CA.

      openssl genrsa -out "relay-int.key" 2048
      
  3. Generate the certificate signing request (CSR).

      openssl req -new -key "relay-int.key" -out relay-int.csr -subj "/CN=relay-int"
      
  4. Sign the CSR with the root CA key.

      openssl x509 -req \
     -days 3650 \
     -CA relay-root-ca.crt -CAkey relay-root-ca.key \
     -set_serial 0 \
     -in relay-int.csr -out relay-int.crt \
     -extensions v3_req -extfile "relay-intermediate-ca.conf"
      
  5. Save the intermediate CA certificate and key in the relay-tls-signing-secret Kubernetes secret on the management cluster.

      kubectl create secret generic relay-tls-signing-secret -n gloo-mesh \
     --from-file=tls.crt=relay-int.crt \
     --from-file=tls.key=relay-int.key \
     --from-file=ca.crt=relay-root-ca.crt \
     --context ${MGMT_CONTEXT} 
      

Create the server TLS certificate

Use the root CA credentials to generate a TLS certificate for the Gloo management server.

  1. Create the configuration for the server TLS certificate with the *.gloo-mesh wildcard DNS name. Because a wildcard is used, the same certificate can later be used to configure the Gloo telemetry gateway.
      cat >"relay-server.conf" <<EOF
    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth, serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS = *.gloo-mesh
    EOF
      
  2. Generate the private key for the Gloo management server.
      openssl genrsa -out "relay-server.key" 2048
      
  3. Generate the certificate signing request for the Gloo management server.
      openssl req -new -key "relay-server.key" -out "relay-server.csr" -subj "/CN=*.gloo-mesh" -config "relay-server.conf"
      
  4. Use the root CA credentials that you created earlier to sign the certificate signing request and create the server TLS certificate.
      openssl x509 -req \
     -days 3650 \
     -CA relay-root-ca.crt -CAkey relay-root-ca.key \
     -set_serial 0 \
     -in relay-server.csr -out relay-server.crt \
     -extensions v3_req -extfile "relay-server.conf"
      
  5. Store the server TLS certificate, private key, and root CA in the relay-server-tls-secret Kubernetes secret.
      kubectl create secret generic relay-server-tls-secret -n gloo-mesh \
     --from-file=tls.crt=relay-server.crt \
     --from-file=tls.key=relay-server.key \
     --from-file=ca.crt=relay-root-ca.crt \
     --context ${MGMT_CONTEXT} 
      

Create the telemetry pipeline credentials

  1. Use the same credentials for the Gloo telemetry gateway and store them in the gloo-telemetry-gateway-tls-secret Kubernetes secret. Using the same credentials is possible, because the configuration for the Gloo management server (relay-server.conf) used a wildcard for the DNS name.
      kubectl create secret generic gloo-telemetry-gateway-tls-secret -n gloo-mesh \
     --from-file=tls.crt=relay-server.crt \
     --from-file=tls.key=relay-server.key \
     --from-file=ca.crt=relay-root-ca.crt \
     --context ${MGMT_CONTEXT} 
      
  2. Store the root CA certificate in the telemetry-root-secret Kubernetes secret on the management and each workload cluster so that the Gloo telemetry collector agent can verify the identity of the Gloo telemetry gateway.
      kubectl create secret generic telemetry-root-secret \
     --from-file=ca.crt=relay-root-ca.crt \
     --namespace gloo-mesh \
     --context ${MGMT_CONTEXT} 
      
      kubectl create secret generic telemetry-root-secret \
     --from-file=ca.crt=relay-root-ca.crt \
     --namespace gloo-mesh \
     --context ${REMOTE_CONTEXT1} 
      

Create relay identity tokens

To establish initial trust between the Gloo management server and agent, you must set up your own relay identity tokens.

  1. Create an environment variable with your identity token. The token can be any string value.

      export TOKEN="<identity_token>"
      
  2. Store the token in the relay-identity-token-secret Kubernetes secret on the management cluster.

      kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context "$MGMT_CONTEXT" --from-literal=token=$TOKEN
      
  3. Copy the identity token to each workload cluster that you want to register.

      kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context $REMOTE_CONTEXT1 --from-literal=token=$TOKEN
    kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context $REMOTE_CONTEXT2 --from-literal=token=$TOKEN
      

Install Gloo Mesh Core

  1. Follow the Install with Helm guide to set up Gloo Mesh Core.
  2. In your Helm values file for the management server, add the following values. Note that mTLS is the default mode in Gloo Mesh Core and does not require any additional configuration.
      
    glooMgmtServer:
      enabled: true
      relay:
        disableCaCertGeneration: true
        signingTlsSecret:
          name: relay-tls-signing-secret
          namespace: gloo-mesh
        tlsSecret:
          name: relay-server-tls-secret
          namespace: gloo-mesh
        tokenSecret:
          key: token
          name: relay-identity-token-secret
          namespace: gloo-mesh
    telemetryCollector: 
      enabled: true
      extraVolumes: 
        - name: root-ca
          secret:
            defaultMode: 420
            optional: true
            secretName: telemetry-root-secret
        - configMap:
            items:
              - key: relay
                path: relay.yaml
            name: gloo-telemetry-collector-config
          name: telemetry-configmap
        - hostPath:
            path: /var/run/cilium
            type: DirectoryOrCreate
          name: cilium-run
    telemetryGateway:
      enabled: true
      extraVolumes:
      - name: tls-keys
        secret:
          secretName:  gloo-telemetry-gateway-tls-secret
          defaultMode: 420
      - name: telemetry-configmap
        configMap:
          name: gloo-telemetry-gateway-config
          items:
            - key: relay
              path: relay.yaml
    telemetryGatewayCustomization:
      disableCertGeneration: true
      
    Helm valueDescription
    glooMgmtServer.relay.disableCaCertGenerationDisable the generation of self-signed certificates to secure the relay connection between the Gloo management server and agent.
    glooMgmtServer.relay.signingTlsSecretAdd the name and namespace of the Kubernetes secret that holds the intermediate CA credentials that you created earlier.
    glooMgmtServer.relay.tlsSecretAdd the name and namespace of the Kubernetes secret that holds the server TLS certificate for the Gloo management server that you created earlier.
    glooMgmtServer.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you created earlier.
    telemetryGateway.extraVolumesAdd the gloo-telemetry-gateway-tls-secret-custom Kubernetes secret that you created earlier to the tls-keys volume. Make sure that you also add the other volumes to your telemetry gateway configuration.
    telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.
  3. In your Helm values file for the agent, add the following values.
      
    glooAgent:
      enabled: true
      relay:
        authority: gloo-mesh-mgmt-server.gloo-mesh
        clientTlsSecretRotationGracePeriodRatio: ""
        rootTlsSecret:
          name: relay-root-tls-secret
          namespace: gloo-mesh
        tokenSecret:
          key: token
          name: relay-identity-token-secret
          namespace: gloo-mesh
    telemetryCollector: 
      enabled: true
      extraVolumes: 
        - name: root-ca
          secret:
            defaultMode: 420
            optional: true
            secretName: telemetry-root-secret
        - configMap:
            items:
              - key: relay
                path: relay.yaml
            name: gloo-telemetry-collector-config
          name: telemetry-configmap
        - hostPath:
            path: /var/run/cilium
            type: DirectoryOrCreate
          name: cilium-run
      
    Helm valueDescription
    glooAgent.relay.rootTlsSecretAdd the name and namespace of the Kubernetes secret that holds the root CA credentials that you copied from the management cluster to the workload cluster.
    glooAgent.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you copied from the management cluster to the workload cluster.