A Gloo Mesh Enterprise setup consists of one management cluster that the Gloo management components are installed in, and one or more workload clusters that run workloads that are registered with and managed by the management cluster. The management cluster serves as the management plane, and the workload clusters serve as the data plane. For more information, see the Relay architecture concept.

Gloo Mesh Enterprise offers several options to secure the relay connection between the Gloo management server and agent. The option that is right for you depends on the type of environment that you have, the security requirements within your organization, and the level of control you want to have over the server and client TLS certificate lifecycle.

Insecure

If you prefer to set up Gloo Mesh Enterprise without secure communication for quick demonstrations, you can install Gloo Mesh Enterprise and register workload clusters in insecure mode. Insecure mode means that no certificates are generated to secure the connection between the Gloo management server and the agent. Instead, they use the HTTP protocol to communicate with each other.

Figure: Insecure connection between the Gloo management server and agents
Figure: Insecure connection between the Gloo management server and agents

For information about how to set up this option, see Insecure setup.

Simple TLS

Secure the relay connection between the Gloo management server and agents by using simple TLS. In a simple TLS setup only the Gloo management server is configured with a server TLS certificate that is used to prove the server’s identity. The identity of the Gloo agent is not verified. To establish initial trust, relay tokens are used.

Review the options that are available to you when you configure simple TLS for the relay connection between the Gloo management server and agent.

Self-signed server TLS certificate

Use self-signed certificates to set up the root CA in the management cluster. The root CA is used to derive the server TLS certificate that the Gloo management server uses to prove its identity to the Gloo agent and encrypt traffic. To establish initial trust between the Gloo management server and agent, relay identity tokens are used that you specify during the Gloo Mesh Enterprise installation.

Figure: TLS connection with self-signed Gloo management server certificate
Figure: TLS connection with self-signed Gloo management server certificate
  1. When you install Gloo Mesh Enterprise, a self-signed root certificate and private key are automatically created for you and stored in the relay-root-tls-secret Kubernetes secret on the management cluster. The secret represents the relay root certificate authority (CA).
  2. Gloo Mesh Enterprise creates a Gloo server TLS certificate that is signed by the root CA. This certificate is stored in the relay-server-tls-secret Kubernetes secret on the management cluster.
  3. Gloo Mesh Enterprise stores the RELAY_TOKEN that you specified in your Helm values file for the management cluster in the relay-identity-token-secret Kubernetes secret on the management cluster.
  4. When you register a cluster with the Gloo management server, the RELAY_TOKEN that you specified in the Helm values file for the agent is stored in memory on the Gloo agent pod. When the Gloo agent tries to connect to the Gloo management server for the first time, the agent sends the RELAY_TOKEN to establish initial trust.
  5. The Gloo management server verifies the relay token that the agent sent by comparing it to the relay token that is stored in the relay-identity-token-secret Kubernetes secret on the management cluster. Only if the relay token values match, the Gloo management server proceeds.
  6. The Gloo management server and agent perform a TLS handshake and establish a TLS connection.

For information about how to set up this option, see Self-signed server certificate.

Bring your own server TLS certificate

Instead of using self-signed certificates for the root CA, you can choose to bring your own certificate management tools and PKI provider to have full control over the server TLS certificate lifecycle. For example, you can use your preferred PKI provider to generate and store the root CA credentials outside your cluster. Then, you use a tool like cert-manager to derive the server TLS certificate and to store it in a Kubernetes secret on the management cluster. Cert-manager monitors the expiration of your certificate and renews it before it expires. To establish initial trust between the Gloo management server and agent, relay identity tokens are used that you specify during the Gloo Mesh Enterprise installation.

Figure: TLS connection with custom Gloo management server certificate
Figure: TLS connection with custom Gloo management server certificate
  1. You use your preferred PKI provider to create a root CA certificate and private key. Note that for security purposes, you might use the root CA to sign a subsequent intermediate CA certificate and key that are also stored in the PKI provider. Then, the intermediate CA certificate and key become the “root” CA certificate and key in the subsequent steps.
  2. You use your preferred PKI provider to create a server TLS certificate for the Gloo management server that is signed by the root CA. You store this certificate in the relay-server-tls-secret-custom Kubernetes secret. You can use a different name for the secret. However, you cannot use relay-server-tls-secret as this name is reserved by Gloo Mesh Enterprise when creating self-signed server TLS certificates.
  3. You use the same credentials that you used for the Gloo management server to configure the Gloo telemetry gateway. Store these credentials in the gloo-telemetry-gateway-tls-secret-custom Kubernetes secret. You can use a different name for the secret. However, you cannot use gloo-telemetry-gateway-tls-secret as this name is reserved by Gloo Mesh Enterprise when creating self-signed server TLS certificates.
  4. To allow Gloo telemetry collector agents to verify the TLS certificate of the Gloo telemetry gateway, you store the root CA certificate in the telemetry-root-secret on the management cluster and each workload cluster.
  5. When you install the Gloo Mesh Enterprise management plane, the Gloo management server is configured with the custom certificate that is stored in the relay-server-tls-secret-custom Kubernetes secret. In addition, the RELAY_TOKEN that you specified in your Helm values file for the management server is stored in the relay-identity-token-secret Kubernetes secret on the management cluster.
  6. When you register a cluster with the Gloo management server, the RELAY_TOKEN that you specified in the Helm values file for the agent is stored in memory on the Gloo agent pod. When the Gloo agent tries to connect to the Gloo management server for the first time, the agent sends the RELAY_TOKEN to establish initial trust.
  7. The Gloo management server verifies the relay token that the agent sent by comparing it to the relay token that is stored in the relay-identity-token-secret Kubernetes secret on the management cluster. Only if the relay token values match, the Gloo management server proceeds.
  8. The Gloo management server and agent perform a TLS handshake and establish a TLS connection.

For information about how to set up this option, see BYO server certificate.

Mutual TLS

In a mutual TLS or two-way TLS setup, both the Gloo management server and agents are set up with TLS certificates. Before establishing a connection, the Gloo management server and agent must present the certificate to each other to prove and validate their identities.

Review the options that are available to you when you configure mTLS for the relay connection between the Gloo management server and agent.

Self-signed CAs with automatic client certificate rotation

In this approach, the Gloo management server automatically generates and signs its own root and intermediate CA certificate and key, and stores these credentials in the management cluster. The root CA is used to derive the server TLS certificate that the Gloo management server uses to prove its identity. In addition, a relay identity token is automatically generated and stored in a Kubernetes secret. When a Gloo agent tries to register with the Gloo management server, the agent must send a matching relay token to establish initial trust. After initial trust is established, the Gloo management server uses the intermediate CA credentials to issue and sign a client TLS certificate that the agent can use in subsequent requests to prove its identity.

Gloo Mesh Enterprise monitors the expiration of the client TLS certificates and automatically renews them before they expire. For more information, see Certificate rotation overview. However, keep in mind that the root CA, intermediate CA, and server TLS certificates are not rotated automatically. You must set up internal processes to monitor the expiration of these certificates.

Figure: Relay connection with Gloo self-signed root CA certificates
Figure: Relay connection with Gloo self-signed root CA certificates

  1. When you install Gloo Mesh Enterprise, a self-signed root certificate and private key are automatically created for you and stored in the relay-root-tls-secret Kubernetes secret on the management cluster. The secret represents the relay root certificate authority (CA).
  2. Gloo Mesh Enterprise then creates an intermediate CA certificate and key that is used to sign certificates for Gloo agents. The certificate and key are stored in the relay-tls-signing-secret Kubernetes secret.
  3. Gloo Mesh Enterprise creates a Gloo server TLS secret that is signed by the root CA. This certificate is stored in the relay-server-tls-secret Kubernetes secret on the management cluster.
  4. Gloo Mesh Enterprise creates an identity token that is used to validate an initial connection between the Gloo agent and the management server. This identity token is stored in the relay-identity-token-secret Kubernetes secret on the management cluster. Note that you cannot bring your own relay identity tokens when using self-signed certificates for the mTLS connection.
  5. Before a workload cluster is registered with the management cluster, the root CA certificate must be added in the relay-root-tls-secret Kubernetes secret on the workload cluster. This secret is automatically set up for you when you use meshctl to register the cluster.
  6. Before a workload cluster is registered with the management cluster, the relay-identity-token-secret Kubernetes secret must be created on the workload cluster. The identity token must match the value of the identity token that is present on the management cluster. To request a client TLS certificate, the Gloo agent uses the identity token as a proof of its identity in the initial TLS connection with the Gloo management server. The identity token is automatically added to the workload cluster when you use meshctl to register the cluster.
  7. The Gloo agent creates a private key and stores this key in the relay-tls-client-secret Kubernetes secret. This key is used to encrypt traffic to the management server.
  8. To start the initial TLS connection to the management server, the Gloo agent gets the value of the identity token from the relay-identity-token-secret Kubernetes secret.
  9. The Gloo agent sends a request to the management server that includes the root CA certificate, the identity token and the private key that it created. This information is used by the management server to verify the agent’s identity and establish initial trust.
  10. The Gloo management server verifies the identity token that was sent by the Gloo agent by comparing it to the identity token that is stored in the relay-identity-token-secret Kubernetes secret on the management cluster.
  11. The Gloo management server uses the private key of the intermediate CA to sign and issue a client TLS certificate.
  12. The client TLS certificate is returned to the Gloo agent.
  13. The Gloo agent stores the TLS certificate in the relay-tls-client-secret on the workload cluster.
  14. For subsequent requests, the Gloo agent uses the client TLS certificate to prove its identity to the Gloo management server and to establish a mutual TLS connection with the server.

For information about how to set up this option, see Self-signed server certificate with managed client certificates.

Bring your own CAs with automatic client TLS certificate rotation

Instead of using self-signed certificates for the root CA, you can use your preferred PKI provider to create the root and intermediate CA certificates and private keys for you. You then store the intermediate CA credentials in a Kubernetes secret on the management cluster so that Gloo Mesh Enterprise can use these credentials to automatically issue client TLS certificates for the agents in your workload clusters. To establish initial trust between the Gloo management server and agent, you must create relay identity tokens and store them in a Kubernetes secret. After the Gloo agent successfully connects, a client TLS certificate is issued that the agent uses in subsequent requests to prove its identity. This setup is recommended for production environments.

Gloo Mesh Enterprise has built-in certificate rotation capabilities that you can leverage to automate the lifecycle of the client TLS certificate. Because the private key of the intermediate CA exists in the cluster, Gloo Mesh Enterprise can monitor and automatically rotate client TLS certificates for the Gloo agent in a workload cluster before they expire. For more information, see Certificate rotation overview. However, keep in mind that the root CA and server TLS certificates are not rotated automatically. You must set up internal processes to monitor the expiration of these certificates.

Figure: Custom root and intermediate CAs when using an identity token to establish initial trust
Figure: Custom root and intermediate CAs when using an identity token to establish initial trust

  1. You use your preferred PKI provider to create a root CA certificate and private key. Note that for security purposes, you might use the root CA to sign a subsequent intermediate CA certificate and key that are also stored in the PKI provider. Then, the intermediate CA certificate and key become the “root” CA certificate and key in the subsequent steps.
  2. You store the root CA certificate in the relay-root-tls-secret on the management cluster and on each workload cluster.
  3. You use your preferred PKI provider to create an intermediate CA certificate and key that is signed by the root CA. You store these credentials in the relay-tls-signing-secret Kubernetes secret on the management cluster.
  4. You use your preferred PKI provider to create a server TLS certificate for the Gloo management server that is signed by the root CA. You store this certificate in the relay-server-tls-secret Kubernetes secret.
  5. You use the same credentials that you used for the Gloo management server to configure the Gloo telemetry gateway. Store these credentials in the gloo-telemetry-gateway-tls-secret Kubernetes secret.
  6. To allow Gloo telemetry collector agents to verify the TLS certificate of the Gloo telemetry gateway, you store the root CA certificate in the telemetry-root-secret on the management cluster and each workload cluster.
  7. You create an identity token of your choice and store that in the relay-identity-token-secret Kubernetes secret on the management cluster and each workload cluster. You can use any string value that you want, such as 1vafj9032npvr90n20fn.
  8. The Gloo agent creates a private key and stores this key in the relay-tls-client-secret Kubernetes secret. This key is used to encrypt traffic to the management server.
  9. The Gloo agent gets the value of the identity token from the relay-identity-token-secret Kubernetes secret and the root CA certificate from the relay-root-tls-secret secret.
  10. The Gloo agent sends a request to the management server that includes the root CA certificate, the identity token and the private key that it created. This information is used by the management server to verify the agent’s identity and establish initial trust.
  11. The Gloo management server verifies the agent’s identity by verifying the identity token and root CA certificate.
  12. After successful verification, the Gloo management server uses the private key of the intermediate CA to sign and issue a client TLS certificate.
  13. The client TLS certificate is returned to the Gloo agent.
  14. The Gloo agent stores the TLS certificate in the relay-tls-client-secret on the workload cluster.
  15. For subsequent requests, the Gloo agent uses the client TLS certificate to prove its identity to the Gloo management server and to establish a mutual TLS connection with the server.

For information about how to set up this option, see BYO server certificate with managed client certificate.

Bring your own CAs and client TLS certificates

To enhance the security of your setup even further and have full control over the root and intermediate CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You then have the option to manually generate the server and client TLS certificates for the Gloo management server and agents and store them in a Kubernetes secret, or to use a tool, such as cert-manager, to automatically issue and rotate client TLS certificates for you. When the Gloo agent tries to connect to the Gloo management server for the first time, the agent presents the client TLS certificate to prove its identity. Because a client TLS certificate is present during the initial connection between the Gloo management server and the agent, no relay identity tokens are required in this setup. This setup is recommended for production.

Figure: Manually provide and rotate Gloo management server and agent certificates
Figure: Manually provide and rotate Gloo management server and agent certificates

  1. You use your preferred PKI provider to create a root CA certificate and private key. Note that for security purposes, you might use the root CA to sign a subsequent intermediate CA certificate and key that are also stored in the PKI provider. Then, the intermediate CA certificate and key become the “root” CA certificate and key in the subsequent steps.
  2. You use the root or intermediate CA in your preferred PKI provider to create a server TLS certificate and key for the Gloo management server. You store this secret in the relay-server-tls-secret Kubernetes secret on the management cluster. Note that you must also add the certificate chain with the root and optionally the intermediate CA certificate.
  3. You use the root or intermediate CA in your preferred PKI provider to create a client TLS certificate and key for the Gloo agent. You store this secret in the relay-client-tls-secret Kubernetes secret on the workload cluster. Note that you must also add the certificate chain with the root and optionally the intermediate CA certificate.
  4. You use the same credentials that you used for the Gloo management server to configure the Gloo telemetry gateway. Store these credentials in the gloo-telemetry-gateway-tls-secret Kubernetes secret.
  5. To allow Gloo telemetry collector agents to verify the TLS certificate of the Gloo telemetry gateway, you store the root CA and optionally the intermediate CA certificates in the telemetry-root-secret on the management cluster and each workload cluster.
  6. The Gloo agent establishes a secure mTLS connection to the management server by using the client TLS certificate.

For information about how to set up this option, see BYO server and client certificates.