☰
Navigation :

Generate your own relay certificates, such as with OpenSSL

To secure communication between the Gloo Mesh management and data planes, set up the relay root and intermediate certificate authorities (CAs) to generate the relay server certificate and relay agent client certificates.

The following steps contain example configurations to generate each relay certificate manually. You can use these example configurations as a starting point to create CAs in your own public key infrastructure (PKI).

Because the CA private key is also stored in the cluster, this method is not recommended for production use and is sufficient for testing only.

Before you begin

  1. Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>. 
    export MGMT_CLUSTER=<mgmt-cluster-name>
export REMOTE_CLUSTER=<remote-cluster-name>
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT=<remote-cluster-context>

  2. To use a custom CA, install cert-manager.

    Example command to install the cert-manager in the management cluster:

    kubectl --context ${MGMT_CONTEXT} apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml

  3. The default openssl version that is included in macOS is LibreSSL, which does not work with these instructions.

    Make sure that you have the OpenSSL version of openssl, not LibreSSL. The openssl version must be at least 1.1.

    1. Check the openssl version that is installed. If you see LibreSSL in the output, continue to the next step. 
      openssl version
    2. Install the OpenSSL version (not LibreSSL). For example, you might use Homebrew. 
      brew install openssl
    3. Review the output of the OpenSSL installation for the path of the binary file. You can choose to export the binary to your path, or call the entire path whenever the following steps use an openssl command.
      • For example, openssl might be installed along the following path: /usr/local/opt/openssl@3/bin/
      • To run commands, you can append the path so that your terminal uses this installed version of OpenSSL, and not the default LibreSSL. /usr/local/opt/openssl@3/bin/openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650...

Generating certificates

Upload your own self-signed CA certificate to the cluster. Because the CA private key is also stored in the cluster, this method is not recommended for production use and is sufficient for testing only.

  1. If it doesn't already exist, create the gloo-mesh namespace in each cluster.

    kubectl create namespace gloo-mesh --context $MGMT_CONTEXT
kubectl create namespace gloo-mesh --context $REMOTE_CONTEXT

  2. Generate the relay root CA.

    1. Generate a relay-root-ca root CA locally.

      openssl req -new -newkey rsa:4096 -x509 -sha256 \
        -days 3650 -nodes -out relay-root-ca.crt -keyout relay-root-ca.key \
        -subj "/CN=relay-root-ca" \
        -addext "extendedKeyUsage = clientAuth, serverAuth"

    2. Upload the root CA to the cluster for use with cert-manager.

      kubectl create secret generic relay-root-ca \
  --from-file=tls.key=relay-root-ca.key \
  --from-file=tls.crt=relay-root-ca.crt \
  --context $MGMT_CONTEXT \
  --namespace gloo-mesh

    3. Create a cert-manager issuer for the CA.

      kubectl --context $MGMT_CONTEXT apply -f -<<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: relay-root-ca
  namespace: gloo-mesh
spec:
  ca:
    secretName: relay-root-ca
EOF

    Generate your certificates locally and only upload the required client/server relay certificates to your cluster.

    openssl req -new -newkey rsa:4096 -x509 -sha256 \
        -days 3650 -nodes -out relay-root-ca.crt -keyout relay-root-ca.key \
        -subj "/CN=relay-root-ca" \
        -addext "extendedKeyUsage = clientAuth, serverAuth"

  3. Generate the gloo-mesh-mgmt-server server certificates.

    Create a cert-manager certificate and refer to the cert-manager generated issuer.

    kubectl --context $MGMT_CONTEXT apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gloo-mesh-mgmt-server
  namespace: gloo-mesh
spec:
  commonName: gloo-mesh-mgmt-server
  dnsNames:
    - "*.gloo-mesh"
  # 1 year life
  duration: 8760h0m0s
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: relay-root-ca
  renewBefore: 8736h0m0s
  secretName: relay-server-tls-secret
  usages:
    - server auth
    - client auth
  privateKey:
    algorithm: "RSA"
    size: 4096
EOF

    Complete the following substeps.

    1. Create the gloo-mesh-mgmt-server certificate configuration. 
      # Server certificate configuration
cat > "gloo-mesh-mgmt-server.conf" <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh
EOF
    2. Generate the private key and certificate signing request (CSR). 
      # Generate gloo-mesh-mgmt-server private key
openssl genrsa -out "gloo-mesh-mgmt-server.key" 2048
# Generate gloo-mesh-mgmt-server CSR
openssl req -new -key "gloo-mesh-mgmt-server.key" -out gloo-mesh-mgmt-server.csr -subj "/CN=gloo-mesh-mgmt-server" -config "gloo-mesh-mgmt-server.conf"
    3. Sign the CSR with the relay-root-ca. 
      # Sign certificate with local relay-root-ca
openssl x509 -req \
  -days 3650 \
  -CA relay-root-ca.crt -CAkey relay-root-ca.key \
  -set_serial 0 \
  -in gloo-mesh-mgmt-server.csr -out gloo-mesh-mgmt-server.crt \
  -extensions v3_req -extfile "gloo-mesh-mgmt-server.conf"
    4. Save the certificate in a Kubernetes secret in the cluster. 
      kubectl create secret generic relay-server-tls-secret \
  --from-file=tls.key=gloo-mesh-mgmt-server.key \
  --from-file=tls.crt=gloo-mesh-mgmt-server.crt \
  --from-file=ca.crt=relay-root-ca.crt \
  --context ${MGMT_CONTEXT} \
  --namespace gloo-mesh

  4. Generate an gloo-mesh-agent client certificate for each workload cluster. Be sure to repeat these steps for each workload cluster that you plan to register with Gloo Mesh.

    1. Create a cert-manager certificate and refer to the cert-manager generated issuer.

      CLUSTER_NAME=$REMOTE_CLUSTER
kubectl apply --context $MGMT_CONTEXT -f - << EOF
kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
  name: gloo-mesh-agent-$CLUSTER_NAME
  namespace: gloo-mesh
spec:
  commonName: gloo-mesh-agent-$CLUSTER_NAME
  dnsNames:
    # Must match the cluster name used in the helm chart install
    - "$CLUSTER_NAME"
  # 1 year life
  duration: 8760h0m0s
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: relay-root-ca
  renewBefore: 8736h0m0s
  secretName: gloo-mesh-agent-$CLUSTER_NAME-tls-cert
  usages:
    - digital signature
    - key encipherment
    - client auth
    - server auth
  privateKey:
    algorithm: "RSA"
    size: 4096
EOF

    2. Copy the TLS secret to the workload cluster.

      kubectl get secret gloo-mesh-agent-$REMOTE_CLUSTER-tls-cert \
  --namespace gloo-mesh \
  --output json \
  --context $MGMT_CONTEXT \
  | jq 'del(.metadata.creationTimestamp,.metadata.resourceVersion,.metadata.uid)' \
  | kubectl apply --context $CLUSTER_CONTEXT -f -

    Complete the following substeps for each remote workload cluster. If the management cluster is also registered as a remote workload cluster, repeat the steps for that cluster, too.

    1. Set the environment variable for your workload cluster. 
      CLUSTER_NAME=$REMOTE_CLUSTER
CLUSTER_CONTEXT=$REMOTE_CONTEXT
    2. Create the gloo-mesh-agent certifiate configuration. 
      # DNS = $CLUSTER_NAME must match the clusterName used in the gloo-mesh-agent install !!
echo "[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = $CLUSTER_NAME" > "gloo-mesh-agent-$CLUSTER_NAME.conf"
    3. Generate the private key and certificate signing request (CSR). 
      # Generate private key
openssl genrsa -out "gloo-mesh-agent-$CLUSTER_NAME.key" 2048
# Create CSR
openssl req -new -key "gloo-mesh-agent-$CLUSTER_NAME.key" -out gloo-mesh-agent-$CLUSTER_NAME.csr -subj "/CN=gloo-mesh-mgmt-server-ca" -config "gloo-mesh-agent-$CLUSTER_NAME.conf"
    4. Sign the CSR with the relay-root-ca. 
      # Sign certificate with root
openssl x509 -req \
  -days 3650 \
  -CA relay-root-ca.crt -CAkey relay-root-ca.key \
  -set_serial 0 \
  -in gloo-mesh-agent-$CLUSTER_NAME.csr -out gloo-mesh-agent-$CLUSTER_NAME.crt \
  -extensions v3_req -extfile "gloo-mesh-agent-$CLUSTER_NAME.conf"
    5. Save the certificate in a Kubernetes secret in the workload cluster. 
      kubectl create secret generic gloo-mesh-agent-$CLUSTER_NAME-tls-cert \
  --from-file=tls.key=gloo-mesh-agent-$CLUSTER_NAME.key \
  --from-file=tls.crt=gloo-mesh-agent-$CLUSTER_NAME.crt \
  --from-file=ca.crt=relay-root-ca.crt \
  --context $CLUSTER_CONTEXT \
  --namespace gloo-mesh

Verify the cert-manager resources

For clusters that have cert-manager installed, verify that your cert-manager issuer and certificate resources are ready. If the READY column says False for any of the following resources, describe the resource for more details and resolve the issue before continuing.

kubectl get issuer -n gloo-mesh --context $MGMT_CONTEXT
kubectl get certificates -n gloo-mesh --context $MGMT_CONTEXT

Now that your custom certificates are created, continue to the next section to modify your Gloo Mesh deployment to use these certificates.

Modifying the Gloo Mesh installation Helm charts

To use the custom CAs and certificates that you create, you must modify the Gloo Mesh installation and registration Helm charts to use these values instead of the default values, such as the default certificates that are generated and managed by Gloo Mesh.

If you already installed Gloo Mesh via Helm, you can upgrade the Helm installation to use these Helm values instead.

gloo-mesh-enterprise Helm chart

Modify and install the gloo-mesh-enterprise Helm chart in your management cluster.

  1. Prepare the Helm installation settings for the gloo-mesh-enterprise chart. Note that you might need to update the values, depending on your certificate setup. 

    
   # Set to true to permit unencrypted and unauthenticated communication between management plane and data planes.
   insecure: false
   # Name of the management plane cluster.
   global:
      cluster: $MGMT_CLUSTER
   # Set up details for the relay certificates.
   relay:
     # Set to true to disable gloo-mesh-mgmt-server relay CA functionality. 
     # Set to true to disable the default self-signed relay certificates and instead use your own.
     disableCa: true
     # Set to true to disable automatically generating self-signed certificates.
     disableCaCertGeneration: true
     # Reference to a Secret containing TLS Certificates used to secure the Gloo Mesh gRPC relay server with TLS.
     tlsSecret:
       name: relay-server-tls-secret
   
    
   --set glooMeshLicenseKey=${GLOO_MESH_LICENSE_KEY} \
   --set glooMeshMgmtServer.relay.tlsSecret.name=relay-server-tls-secret \
   --set glooMeshMgmtServer.relay.disableCaCertGeneration=true \
   --set glooMeshMgmtServer.relay.disableCa=true

  2. Modify the Gloo Mesh Enterprise Helm chart installation with the updated settings.

gloo-mesh-agent Helm chart

Modify and install the gloo-mesh-agent Helm chart in each workload cluster.

  1. With the Kubernetes context still set to your management cluster, get the IP address of the gloo-mesh-mgmt-server deployment. 

    
MGMT_INGRESS_ADDRESS=$(kubectl get svc -n gloo-mesh gloo-mesh-mgmt-server --context ${MGMT_CONTEXT} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
MGMT_INGRESS_PORT=$(kubectl -n gloo-mesh get service gloo-mesh-mgmt-server --context ${MGMT_CONTEXT} -o jsonpath='{.spec.ports[?(@.name=="grpc")].port}')
MGMT_SERVER_NETWORKING_ADDRESS=${MGMT_INGRESS_ADDRESS}:${MGMT_INGRESS_PORT}

    
MGMT_INGRESS_ADDRESS=$(kubectl get nodes --context ${MGMT_CONTEXT} -ojsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
MGMT_INGRESS_PORT=$(kubectl -n gloo-mesh get service gloo-mesh-mgmt-server --context ${MGMT_CONTEXT} -o jsonpath='{.spec.ports[?(@.name=="grpc")].port}')
MGMT_SERVER_NETWORKING_ADDRESS=${MGMT_INGRESS_ADDRESS}:${MGMT_INGRESS_PORT}

  2. Create a values.yaml file to upgrade your Gloo Mesh agent Helm chart installation. Note that you might need to update the values, depending on your certificate setup.

    
   # Set to true to permit unencrypted and unauthenticated communication between management plane and data planes.
   insecure: false
   relay:
     # SNI name used to connect to relay forwarding server, mustmatch server certificate CommonName (DO NOT CHANGE)
     authority: gloo-mesh-mgmt-server.gloo-mesh
     # gloo-mesh-mgmt-server IP address
     serverAddress: $MGMT_SERVER_NETWORKING_ADDRESS
     # Reference to a secret containing the client TLS certs used to identify the relay agent to the relay server.
     clientTlsSecret:
       name: gloo-mesh-agent-$REMOTE_CLUSTER-tls-cert
       namespace: gloo-mesh
     # Reference to a secret containing a root TLS cert used to verify the relay server cert. 
     # The secret can also optionally specify a 'tls.key' which will be used to generate the agent client cert.
     rootTlsSecret:
       name: relay-root-tls-secret
       namespace: gloo-mesh
   
    
   --set relay.serverAddress=${MGMT_SERVER_NETWORKING_ADDRESS} \
   --set relay.clientTlsSecret.name=gloo-mesh-agent-$REMOTE_CLUSTER-tls-cert \
   --set relay.clientTlsSecret.namespace=gloo-mesh \
   --set relay.rootTlsSecret.name=relay-root-tls-secret \
   --set relay.rootTlsSecret.namespace=gloo-mesh

  3. Modify the Gloo Mesh agent Helm chart installation with the updated settings in your values.yaml file.

  4. Allow the Gloo Mesh management plane to use the relay certificates to connect to the agents. The steps vary depending on whether you are installing Gloo Mesh for the first time, or upgrading an existing installation.

    Create a KubernetesCluster object in the management cluster for your workload cluster.

    kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata:
  name: ${REMOTE_CLUSTER}
  namespace: gloo-mesh
spec:
  clusterDomain: cluster.local
EOF

    Reload the gloo-mesh-mgmt-server and gloo-mesh-agent pods to pick up the new certificates.

    Restarting the Gloo Mesh pods does not impact your running apps. However, you cannot change the configuration of Gloo Mesh resources, such as to control traffic policies, until the pods are healthy again.

    1. Get the name of the gloo-mesh-mgmt-server pod in your management cluster. 
      kubectl get pods -n gloo-mesh --context $MGMT_CONTEXT
    2. Restart the gloo-mesh-mgmt-server pod. 
      kubectl delete pod -n gloo-mesh --context $MGMT_CONTEXT <gloo-mesh-mgmt-server-pod>
    3. Get the name of the gloo-mesh-agent pod in your workload cluster. 
      kubectl get pods -n gloo-mesh --context $REMOTE_CONTEXT
    4. Restart the gloo-mesh-agent pod. 
      kubectl delete pod -n gloo-mesh --context $REMOTE_CONTEXT <gloo-mesh-agent-pod>

  5. Repeat these steps for each workload cluster.

Verifying your relay certificate setup

  1. Check that the relay connection between the management server and workload agents is healthy.
    1. Forward port 9091 of the gloo-mesh-mgmt-server pod to your localhost. 
      kubectl port-forward -n gloo-mesh --context $MGMT_CONTEXT deploy/gloo-mesh-mgmt-server 9091
    2. In your browser, connect to http://localhost:9091/metrics.
    3. In the metrics UI, look for the following lines. If the values are 1, the agents in the workload clusters are successfully registered with the management server. If the values are 0, the agents are not successfully connected. 
      relay_pull_clients_connected{cluster="cluster-1"} 1
relay_pull_clients_connected{cluster="cluster-2"} 1
# HELP relay_push_clients_connected Current number of connected Relay push clients (Relay Agents).
# TYPE relay_push_clients_connected gauge
relay_push_clients_connected{cluster="cluster-1"} 1
relay_push_clients_connected{cluster="cluster-2"} 1
  2. Review the Gloo Mesh UI. Check that the Overall Mesh Status is healthy and that your remote clusters are registered without any configuration issues. 
    meshctl dashboard --kubecontext $MGMT_CONTEXT
  3. If the setup is unsuccessful, continue to Troubleshooting.

Troubleshooting relay certificates

  1. Review the health of your Gloo Mesh pods in the management and remote clusters.

    1. Check that the gloo-mesh-mgmt-server and gloo-mesh-agent pods are running.

      kubectl get pods -n gloo-mesh --context ${MGMT_CONTEXT}
kubectl get pods -n gloo-mesh --context ${REMOTE_CONTEXT}

    2. If the pods are not running, describe the pods and check the State and Last State sections for error messages and reasons why the pod might not be healthy. For example, the following error messages in the gloo-mesh-mgmt-server and gloo-mesh-agent pods indicate that the secret is misnamed or missing. Check the secrets and names, upgrade your Helm installation, and try again.

      • Example error message for gloo-mesh-mgmt-server pod:
      Message:   3 errors occurred:
    * no tls secret found for grpc server: Secret "relay-server-tls-secret" not found
    * could not find forwarding server token: no token secret found: Timeout: failed waiting for *v1.Secret Informer to sync
    * no tls secret found for grpc server: Secret "relay-server-tls-secret" not found
      • Example error message for gloo-mesh-agent pod:
      Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Normal   Created      84m (x25 over 3h28m)   kubelet  Created container gloo-mesh-agent
  Normal   Pulled       84m (x24 over 3h28m)   kubelet  Container image "gcr.io/gloo-mesh/gloo-mesh-agent:1.2.3" already present on machine
  Normal   Started      84m (x25 over 3h28m)   kubelet  Started container gloo-mesh-agent
  Warning  FailedMount  84m                    kubelet  MountVolume.SetUp failed for volume "kube-api-access-zlr9b" : [failed to fetch token: Post "https://kind2-control-plane:6443/api/v1/namespaces/gloo-mesh/serviceaccounts/gloo-mesh-agent/token": read tcp 172.18.0.5:47314->172.18.0.5:6443: use of closed network connection, failed to sync configmap cache: timed out waiting for the condition]
  Warning  FailedMount  84m                    kubelet  MountVolume.SetUp failed for volume "kube-api-access-zlr9b" : [failed to fetch token: Post "https://kind2-control-plane:6443/api/v1/namespaces/gloo-mesh/serviceaccounts/gloo-mesh-agent/token": read tcp 172.18.0.5:57262->172.18.0.5:6443: use of closed network connection, failed to sync configmap cache: timed out waiting for the condition]
  Warning  FailedMount  83m                    kubelet  MountVolume.SetUp failed for volume "kube-api-access-zlr9b" : [failed to fetch token: Post "https://kind2-control-plane:6443/api/v1/namespaces/gloo-mesh/serviceaccounts/gloo-mesh-agent/token": http2: client connection force closed via ClientConn.Close, failed to sync configmap cache: timed out waiting for the condition]
  Warning  BackOff      72s (x522 over 3h28m)  kubelet  Back-off restarting failed container

  2. Check the Kubernetes logs for the gloo-mesh-mgmt-server and gloo-mesh-agent pods in each cluster for errors. Look for errors during the grpc connection.

    • For example, the following error message indicates that the gloo-mesh-mgmt-server load balancer IP address was set incorrectly for the agent during the Helm installation.
    {"level":"warn","ts":"2021-11-02T19:56:42.197Z"caller":"zap/
grpclogger.go:85","msg":"[core]grpcaddrConn.createTransport
failed to connect to {34.145.18106:9900:9900 
gloo-mesh-mgmt-server.gloo-mesh <nil> <nil>}. Err: connection 
error: desc = \"transport: Error while dialing dial tcp: address 
34.145.184.106:9900:9900 too many colons in address\".
    • The following gloo-mesh-agent pod error indicates that you need to follow the steps in ca.crt.
    {"level":"fatal","ts":1640102555.6522746,"msg":"secrets \"relay-root-tls-secret\" not found","version":"1.3.0-beta6","stacktrace":"runtime.main\n\t/usr/local/go/src/runtime/proc.go:255"}
    • The following errors indicate that the server or client TLS certificate is expired. Regenerate the certificate, restart the pods, and try again.
    {"level":"error","ts":1650047047.6682806,"logger":"translator.reconcile-42","caller":"translator/reconciler.go:195","msg":"translation for parent object failed","parent":"istio-ingressgateway-istio-system-cluster1~gloo-mesh~cluster1~internal.gloo.solo.io/v2, Kind=DiscoveredGateway","err":"Gateway istio-ingressgateway.istio-system in cluster cluster1 not found in snapshot.","errVerbose":"Gateway istio-ingressgateway.istio-system in cluster cluster1 not found in snapshot.\n\ttranslator.(*translator).TranslateOutputs.func1:/src/pkg/translator/translator.go:163\n\ttranslator.(*translator).translateParallel:/src/pkg/translator/translator.go:189\n\tsets.(*discoveredGatewaySet).UnsortedList:/src/pkg/api/internal.gloo.solo.io/v2/sets/sets.go:999\n\tsets.(*resourceSet).UnsortedList:/go/pkg/mod/github.com/solo-io/skv2@v0.22.11/contrib/pkg/sets/sets.go:118\n\tsets.(*discoveredGatewaySet).UnsortedList.func1:/src/pkg/api/internal.gloo.solo.io/v2/sets/sets.go:994\n\ttranslator.(*translator).translateParallel.func1:/src/pkg/translator/translator.go:191\n\ttranslator.getValidEastWestIngressGateway:/src/pkg/translator/translator.go:426","stacktrace":"github.com/solo-io/gloo-mesh-enterprise/pkg/translator.(*reconciler).reconcilePrimary.func1\n\t/src/pkg/translator/reconciler.go:195\ngithub.com/solo-io/gloo-mesh-enterprise/pkg/utils/syncutils.(*workQueue).Execute.func1\n\t/src/pkg/utils/syncutils/parallel.go:52"}

    {"level":"info","ts":1650046690.815508,"caller":"grpclog/grpclog.go:37","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc00111c9d0, {TRANSIENT_FAILURE connection error: desc = \"transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-04-15T18:18:10Z is after 2022-04-15T14:28:30Z\"}","system":"grpc","grpc_log":true}

  3. For gloo-mesh-agent pods, make sure that the cluster name matches the registered cluster name.

    1. Check the KubernetesCluster resources in the management cluster to get registered cluster names. 
      kubectl get kubernetesclusters --context $MGMT_CONTEXT
    2. Check that the registered cluster name matches the name in the client certificate that is issued by the root CA, specifically the DNS SAN extension.
    3. If the cluster names do not match, update the KubernetesCluster to have the same name, or re-issue the client certificate with the same name.

  4. If you still have issues, review the Known issues.

Known issues

ca.crt

Although the ca.crt is included in the gloo-mesh-agent certificate secret, the gloo-mesh-agent still expects it to exist separately in the remote cluster. To copy it from the management cluster into the remote clusters, you can run the following command. Make sure to update $CLUSTER_NAME with your remote cluster name.

CLUSTER_NAME=$REMOTE_CLUSTER
CLUSTER_CONTEXT=$REMOTE_CONTEXT

kubectl get secret gloo-mesh-agent-$CLUSTER_NAME-tls-cert \
  --namespace gloo-mesh \
  --output json \
  --context $CLUSTER_CONTEXT \
  | jq 'del(.metadata.creationTimestamp,.metadata.resourceVersion,.metadata.uid,.data."tls.key",.data."tls.crt",.metadata.annotations)' \
  | sed 's/gloo-mesh-agent-$CLUSTER_NAME-tls-cert/relay-root-tls-secret/' \
  | kubectl apply --context $CLUSTER_CONTEXT -f -

Certificate renewal

Currently, gloo-mesh-agent cannot automatically update certificates when they are renewed. You must recreate the gloo-mesh-agent pod in order to pick up the new certificates. For this reason, it is currently recommended that you make the time to live on the relay certificates to be long-lived.