Understand federated trust

Gloo Mesh can help unify the root identity between multiple service mesh installations so any intermediates are signed by the same root certificate authority (CA). Federating the root identity ensures that end-to-end mTLS between clusters and destinations can be established correctly.

Gloo Mesh establishes trust according to the trust model that is defined by the user.

In this guide, you learn how Gloo Mesh helps you automatically establish and manage shared trust across Istio clusters.

Follow this guide to learn about how federated trust works, not to set up your production environment. This guide's setup of Gloo Mesh to manage trust across clusters might not meet the security requirements for your production setup, because the root and intermediate CA certificates are stored locally in the management cluster. For more options with certificate management tools, see Certificate architectures.

Before you begin

Complete a getting started guide to get a demo environment that meets the prerequisites.

  1. Make sure that you have three clusters. In this guide, the cluster names mgmt-cluster, cluster-1, and cluster-2 are used. The mgmt-cluster serves as the management cluster, and cluster-1 and cluster-2 serve as the workload clusters in this setup.
  2. Install Gloo Mesh Enterprise and register workload cluster.
  3. Create a workspace that allows federation across clusters.
  4. Install Istio in the workload clusters, cluster-1 and cluster-2, including the bookinfo sample app.
  5. Set the names of your clusters from your infrastructure provider. If your clusters have different names, specify those names instead.
    export MGMT_CLUSTER=mgmt-cluster
    export REMOTE_CLUSTER1=cluster-1
    export REMOTE_CLUSTER2=cluster-2
  6. Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
    export MGMT_CONTEXT=<management-cluster-context>
    export REMOTE_CONTEXT1=<remote-cluster-1-context>
    export REMOTE_CONTEXT2=<remote-cluster-2-context>

Enforce mTLS with Istio

To enforce mutual TLS connections for the workloads in your Istio service mesh, you can apply the following peer authentication configuration to both workload clusters. Update the values as needed, such as if you run istiod in a different namespace than istio-system.

For more information about this Istio setting, see the Istio documentation.

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
  name: "default"
  namespace: "istio-system"
    mode: STRICT

kubectl apply --context $REMOTE_CONTEXT1 -f - << EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
  name: "default"
  namespace: "istio-system"
    mode: STRICT
kubectl apply --context $REMOTE_CONTEXT2 -f - << EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
  name: "default"
  namespace: "istio-system"
    mode: STRICT

Verify separate cluster identities

Review the certificate chain that is used to establish mTLS between Istio destinations in cluster-1 and cluster-2, to verify that the certificates are different. To see the certificates, you can use the openssl s_client tool with the -showcerts parameter when calling between two destinations.

  1. Show the certificates for a pod in cluster-1.

    kubectl --context $REMOTE_CONTEXT1 -n bookinfo exec -it deploy/reviews-v1 -c istio-proxy \
    -- openssl s_client -showcerts -connect ratings.bookinfo:9080
  2. Review the output of the certificate chain among other handshake-related information. The last certificate in the chain is the root certificate.

    Certificate chain
     0 s:
       i:O = cluster.local
    -----END CERTIFICATE-----
     1 s:O = cluster.local
       i:O = cluster.local
    -----END CERTIFICATE-----

  3. Repeat the command in cluster-2. For the reviews service running in cluster-2, use deploy/reviews-v3, because reviews-v1 from the previous command doesn't exist in cluster-2.

    kubectl --context $REMOTE_CONTEXT2 -n bookinfo exec -it deploy/reviews-v3 -c istio-proxy \
    -- openssl s_client -showcerts -connect ratings.bookinfo:9080
  4. Compare the output of the previous command for cluster-2 with the output from the command for cluster-1. Notice that the root certificates that signed the workload certificates are different.

In the next section, you learn how to unify the different certificates into a shared trust model of identity.

Set up shared trust with Gloo Mesh

As the platform admin for your Gloo Mesh environment, you can create a RootTrustPolicy custom resource. The root trust policy sets up the trust domain and root certificates to unify the service meshes by establishing a shared trust model for identity.

  1. Save a basic configuration file for a root trust policy custom resource. The subsequent steps walk you through updating the configuration.

    apiVersion: admin.gloo.solo.io/v2
    kind: RootTrustPolicy
      name: istio-ingressgateway
      namespace: gloo-mesh
        autoRestartPods: true
          generated: {}
  2. Specify details for the service meshes that you want to include for shared trust. The following example selects all service meshes in registered clusters with a label of trust: shared and where istiod runs in the istio-system namespace. If you omit applyToMeshes, all registered service meshes are included by default.

            - trust: shared
          namespace: istio-system
  3. Prepare the root certificate for shared trust. You can optionally skip this step to let Gloo Mesh generate a root certificate for you.

    1. Identify the root certificate file root-cert.pem and its associated private key file key.pem that you want to use. If you need to create these files, see the Istio documentation.
    2. Create the secret for the virtual mesh to use. Note that the name/namespace of the provided root cert cannot be cacerts/istio-system, because that is used by Gloo Mesh for carrying out the CSR (certificate signing request) procedure.
      kubectl -n default create secret generic providedrootcert --from-file=root-cert.pem --from-file=key.pem --context $MGMT_CONTEXT
  4. Set up the configuration for the root CA to use. Note that you can let Gloo Mesh generate the root cert for you by setting the generated field to an empty selector.

               name: $SECRET
               namespace: $NAMESPACE
             generated: {}

  5. Decide if you want to automatically restart Istio workload pods to pick up the certificates. If you set this value to false, you must manually restart your Istio workloads before they can communicate with each other.

    This example includes the autoRestartPods: true setting. Gloo Mesh will restart all of the Istio workloads in all of the cluster, to speed up the certificate rotation for the workloads. To avoid downtime, do NOT set this field to true in production environments.

       autoRestartPods: true
  6. Apply the root trust policy to your management cluster.

    kubectl --context $MGMT_CONTEXT apply -f demo-virtual-mesh.yaml
  7. Wait a few minutes for the Istio workloads to be reloaded to pick up the new certificates.

  8. Verify that the certificates were issued in your workload clusters.

    1. Check that the IssuedCertificates custom resource is created in cluster-1. You can repeat this step for each workload cluster.

      kubectl get issuedcertificates -n gloo-mesh --context $REMOTE_CONTEXT1

      Example output:

      NAME                            AGE
      istiod-istio-system-cluster-2   3m15s
    2. Verify that the cacerts is created with the Gloo Mesh issued certificates. You can repeat this step for each workload cluster.

    kubectl get secret -n istio-system cacerts --context $REMOTE_CONTEXT1

    Example output:

    NAME      TYPE                                               DATA   AGE
    cacerts   certificates.mesh.gloo.solo.io/issued_certificate  5      20s

Now, the connections among the federated destinations in your service meshes share a root identity for shared trust!

Multicluster mesh federation

After shared trust is established, Gloo Mesh can federate Destinations (your Istio services) across clusters so that they can communicate with each other securely. Behind the scenes, Gloo Mesh handles the networking, such as through egress and ingress gateways and as shifted by your user-defined traffic and access policies, to ensure that requests to the service are resolved and routed to the right destination.

You can configure federation in the workspace settings.

To check federated services, you can review the Istio ServiceEntry objects that are created in your clusters.

Notice how the output of the following command includes services that are federated from your other cluster, cluster-2.

kubectl get serviceentry -n istio-system --context $REMOTE_CONTEXT1
NAME                                                          HOSTS                                                           LOCATION        RESOLUTION   AGE
istio-ingressgateway.istio-system.svc.cluster-2.global        [istio-ingressgateway.istio-system.svc.cluster-2.global]        MESH_INTERNAL   DNS          6m2s
ratings.bookinfo.svc.cluster-2.global                         [ratings.bookinfo.svc.cluster-2.global]                         MESH_INTERNAL   DNS          6m2s
reviews.bookinfo.svc.cluster-2.global                         [reviews.bookinfo.svc.cluster-2.global]                         MESH_INTERNAL   DNS          6m2s

Notice how the output of the following command includes services that are federated from your other cluster, cluster-1.

kubectl get serviceentry -n istio-system --context $REMOTE_CONTEXT2
NAME                                                        HOSTS                                                             LOCATION        RESOLUTION   AGE
details.bookinfo.svc.cluster-1.global                       [details.bookinfo.svc.cluster-1.global]                       MESH_INTERNAL   DNS          2m5s     
istio-ingressgateway.istio-system.svc.cluster-1.global      [istio-ingressgateway.istio-system.svc.cluster-1.global]      MESH_INTERNAL   DNS          5m18s    
productpage.bookinfo.svc.cluster-1.global                   [productpage.bookinfo.svc.cluster-1.global]                   MESH_INTERNAL   DNS          55s      
ratings.bookinfo.svc.cluster-1.global                       [ratings.bookinfo.svc.cluster-1.global]                       MESH_INTERNAL   DNS          7m2s     
reviews.bookinfo.svc.cluster-1.global                       [reviews.bookinfo.svc.cluster-1.global]                       MESH_INTERNAL   DNS          90s 

Next steps

Now, you can route traffic across your clusters with end-to-end mTLS and verify that the certificates use a shared root of trust.

You might try integrating a certificate manager provider, such as AWS or Vault.