Navigation :

Understand federated trust

Gloo Mesh can help unify the root identity between multiple service mesh installations so any intermediates are signed by the same root certificate authority (CA). Federating the root identity ensures that end-to-end mTLS between clusters and destinations can be established correctly.

Gloo Mesh establishes trust according to the trust model that is defined by the user.

Shared trust: Is there complete shared trust and a common root and identity?
Limited trust: Is there limited trust between clusters, with traffic gated by egress and ingress gateways?

In this guide, you learn how Gloo Mesh helps you automatically establish and manage shared trust across Istio clusters.

Istio root certificates are currently not automatically rotated in Gloo Mesh Enterprise before they expire. If you created a root trust policy in your cluster, and your root certificate is about to expire, contact your Solo account representative to help with manually rotating the root certificate in your cluster.

Follow this guide to learn about how federated trust works, not to set up your production environment. This guide's setup of Gloo Mesh to manage trust across clusters might not meet the security requirements for your production setup, because the root and intermediate CA certificates are stored locally in the management cluster. For more options with certificate management tools, see Certificate architectures.

Before you begin

Complete a getting started guide to get a demo environment that meets the prerequisites.

Make sure that you have three clusters. In this guide, the cluster names mgmt, cluster1, and cluster2 are used. The mgmt cluster serves as the management cluster, and cluster1 and cluster2 serve as the workload clusters in this setup.
Install Gloo Mesh Enterprise and register workload cluster.
Create a workspace that allows federation across clusters.
Install Istio in the workload clusters, cluster1 and cluster2, including the bookinfo sample app.
Set the names of your clusters from your infrastructure provider. If your clusters have different names, specify those names instead.
```
export MGMT_CLUSTER=mgmt
export REMOTE_CLUSTER1=cluster1
export REMOTE_CLUSTER2=cluster2
```
Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
```
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT1=<remote-cluster1-context>
export REMOTE_CONTEXT2=<remote-cluster2-context>
```

Enforce mTLS with Istio

To enforce mutual TLS connections for the workloads in your Istio service mesh, you can apply the following peer authentication configuration to both workload clusters. Update the values as needed, such as if you run istiod in a different namespace than istio-system.

For more information about this Istio setting, see the Istio documentation.

YAML file
CLI inline


apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT


kubectl apply --context $REMOTE_CONTEXT1 -f - << EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF
kubectl apply --context $REMOTE_CONTEXT2 -f - << EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
EOF

Verify separate cluster identities

Review the certificate chain that is used to establish mTLS between Istio destinations in cluster1 and cluster2, to verify that the certificates are different. To see the certificates, you can use the openssl s_client tool with the -showcerts parameter when calling between two destinations.

Show the certificates for a pod in cluster1.

kubectl --context $REMOTE_CONTEXT1 -n bookinfo exec -it deploy/reviews-v1 -c istio-proxy \
-- openssl s_client -showcerts -connect ratings.bookinfo:9080

Review the output of the certificate chain among other handshake-related information. The last certificate in the chain is the root certificate.

---
Certificate chain
 0 s:
   i:O = cluster.local
-----BEGIN CERTIFICATE-----
MIIDKTCCAhGgAwIBAgIRAKZzWK3r7aZqVd0pXUalzKIwDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMDA0MjMxMjM2MDFaFw0yMDA0
MjQxMjM2MDFaMAAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCwCZft
3uavGRCv+ooKVWUod7Z3PWukPGIR0icI12+ghFygT9ZKlyu+LQ8iN93A6/soIo8r
ccp5YCyW9O4JCJSPg+iFqGeg9yNDLCATb+6vwTsHx0rvdLdX4803bjF9evWkr5yZ
AlPath6S/Wxihue2xrnw9mSF1nKRQxxw8ypysKiqLfVNBhCnBsN28gppYnl1pIiv
YamBeSiNA887BDnuXIc6t6yNJudlvefuixUhzBeR9zYNlstWBLsdqSubbPdPVxfv
7H6hRjeAmu0VB2oDpsWJ0OYGu42ZavCSHRIL2nD3fqk9DyWtXKIklU4B7rE4mySe
InDLbkj1lLv1FB1VAgMBAAGjgYUwgYIwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMEMGA1UdEQEB/wQ5
MDeGNXNwaWZmZTovL2NsdXN0ZXIubG9jYWwvbnMvZGVmYXVsdC9zYS9ib29raW5m
by1yYXRpbmdzMA0GCSqGSIb3DQEBCwUAA4IBAQBEUi1ge/M3NlQ6xuizY7o/mkFe
+PXKjKT/vf21d6N5FTAJT4fjL6nEsa4NqJC7Txiz9kEjlqLy/SywtB3qYGuG0/+d
QGgWmN1NVOMtl2Kq++LOQOIaEV24mjHb5r38DVk4YyVH2E/1QAWByONDB54Ovlyf
l3qiE3gEeegKsgtsLuhzQCReU5evdmPhnCAMiZvUhQKxHIoCJEx5A+eB4q2zBDN2
H2CNJyWLPulBNCsZvCYXGLDRIy+Sp9AsXhqMTAxvqNS2NaNQ9fh7gSqOORO3kIKz
axoFg6neo+LAaYwoyBtO7/V9OvShd9TbkyPP4amR7k3zkdulFo2o+jKAqzCq
-----END CERTIFICATE-----
 1 s:O = cluster.local
   i:O = cluster.local
-----BEGIN CERTIFICATE-----
MIIC3TCCAcWgAwIBAgIQDZ3lILg70fkKSuuBpj3O7TANBgkqhkiG9w0BAQsFADAY
MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIwMDQyMjIzMjM0NVoXDTMwMDQy
MDIzMjM0NVowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBALX8anGTKtlpdbIlwGsxTW/ZJeqSM29eei5Lmsee
wll7xaNE4sNaj6HFyqAZomPDJm/4PYZ0fWmJ1FIXFqCXQ6PNf/J592D1x8oIHh50
88BbOkH7wYzEMymoP+2BqXQsY5kxjCg9N6xj4XygSunjXo3ctyVP11GhUew0j+Aw
U7dtZqWlpgMsZsPEn2V4JFid20q+0qz6iCzRh/a3iO98QSfvlpeQuVQkhLiPZOzA
q796C1HLWU7sefkXzVAsQGHA5FqSQLQbOqXBPWaf82Fw9cO4/skBH/qOIVtIh8Ks
rHMgrYkSXprev4bMafUAdfJ9GLity/4D2Mn0rK3k4GiLoL8CAwEAAaMjMCEwDgYD
VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB
AGZVlJzyM9E14eucxf1UmJvt5NeGGmgf8YdUd9R693iffG8eAg44VJP6xZ8LTrj7
WoGoWC9SmclvAm2Lo1zh7JQWL2jca5X/aJSW4CZROGDMwWm9e+SaujsOKG3hhis6
iwTl1VsjV4o0UBRm5Z26T/gn1CoIPjQDJRb86RPr/6DHY8jFGvGjceEl+o6mf+gk
Q0xfk7VNxpxScJ/+lU5+IJrqQTBmrhk40eDe24D4zbtnk4YVRRbiMh4p9PIBySyp
gyMylEJ3SgwpVoWwV0e2UvNCG1AlZADiYPpgy2qANzJqtF/GYjfgcpR01r8LceIj
s2rL2u8nTerM5bjlurn1Z58=
-----END CERTIFICATE-----

Repeat the command in cluster2. For the reviews service running in cluster2, use deploy/reviews-v3, because reviews-v1 from the previous command doesn't exist in cluster2.
```
kubectl --context $REMOTE_CONTEXT2 -n bookinfo exec -it deploy/reviews-v3 -c istio-proxy \
-- openssl s_client -showcerts -connect ratings.bookinfo:9080
```
Compare the output of the previous command for cluster2 with the output from the command for cluster1. Notice that the root certificates that signed the workload certificates are different.

In the next section, you learn how to unify the different certificates into a shared trust model of identity.

Set up shared trust with Gloo Mesh

As the platform admin for your Gloo Mesh environment, you can create a RootTrustPolicy custom resource. The root trust policy sets up the trust domain and root certificates to unify the service meshes by establishing a shared trust model for identity.

Save a basic configuration file for a root trust policy custom resource. The subsequent steps walk you through updating the configuration.

apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    autoRestartPods: true
    intermediateCertOptions: {}
    mgmtServerCa:
      generated:
        orgName: root-certificate-a

Specify details for the service meshes that you want to include for shared trust. The following example selects all service meshes in registered clusters with a label of trust: shared and where istiod runs in the istio-system namespace. If you omit applyToMeshes, all registered service meshes are included by default.
```
spec:
  applyToMeshes:
    istio:
      clusterSelector:
        - trust: shared
      namespace: istio-system
```
Prepare the root certificate for shared trust. You can optionally skip this step to let Gloo Mesh generate a root certificate for you.
1. Identify the root certificate file root-cert.pem and its associated private key file key.pem that you want to use. If you need to create these files, see the Istio documentation.
2. Create the secret for the virtual mesh to use. Note that the name/namespace of the provided root cert cannot be cacerts/istio-system, because that is used by Gloo Mesh for carrying out the CSR (certificate signing request) procedure.
```
kubectl -n default create secret generic providedrootcert --from-file=root-cert.pem --from-file=key.pem --context $MGMT_CONTEXT
```
Set up the configuration for the root CA to use. Note that you can let Gloo Mesh generate the root cert for you by setting the generated field to an empty selector.
- Use the root cert you prepared
- Let Gloo Mesh generate a root cert
```
   spec:
     config:
       mgmtServerCa:
         secretRef:
           name: $SECRET
           namespace: $NAMESPACE
   
```
```
   spec:
     config:
       mgmtServerCa:
         generated: {}
   
```
Decide if you want to automatically restart Istio workload pods to pick up the certificates. If you set this value to false, you must manually restart your Istio workloads before they can communicate with each other.
This example includes the autoRestartPods: true setting. Gloo Mesh will restart all of the Istio workloads in all of the cluster, to speed up the certificate rotation for the workloads. To avoid downtime, do NOT set this field to true in production environments.
```
spec:
 config:
   autoRestartPods: true
```

Apply the root trust policy to your management cluster.

kubectl --context $MGMT_CONTEXT apply -f demo-virtual-mesh.yaml

Wait a few minutes for the Istio workloads to be reloaded to pick up the new certificates.

Verify that the certificates were issued in your workload clusters.

Check that the IssuedCertificates custom resource is created in cluster1. You can repeat this step for each workload cluster.

kubectl get issuedcertificates -n gloo-mesh --context $REMOTE_CONTEXT1

Example output:

NAME                            AGE
istiod-istio-system-cluster2   3m15s

Verify that the cacerts is created with the Gloo Mesh issued certificates. You can repeat this step for each workload cluster.

kubectl get secret -n istio-system cacerts --context $REMOTE_CONTEXT1

Example output:

NAME      TYPE                                               DATA   AGE
cacerts   certificates.mesh.gloo.solo.io/issued_certificate  5      20s

Now, the connections among the federated destinations in your service meshes share a root identity for shared trust!

Multicluster mesh federation

After shared trust is established, Gloo Mesh can federate Destinations (your Istio services) across clusters so that they can communicate with each other securely. Behind the scenes, Gloo Mesh handles the networking, such as through egress and ingress gateways and as shifted by your user-defined traffic and access policies, to ensure that requests to the service are resolved and routed to the right destination.

You can configure federation in the workspace settings.

To check federated services, you can review the Istio ServiceEntry objects that are created in your clusters.

Cluster 1
Cluster 2

Notice how the output of the following command includes services that are federated from your other cluster, cluster2.

kubectl get serviceentry -n istio-system --context $REMOTE_CONTEXT1

NAME                                                         HOSTS                                                          LOCATION        RESOLUTION   AGE
istio-ingressgateway.istio-system.svc.cluster2.global        [istio-ingressgateway.istio-system.svc.cluster2.global]        MESH_INTERNAL   DNS          6m2s
ratings.bookinfo.svc.cluster2.global                         [ratings.bookinfo.svc.cluster2.global]                         MESH_INTERNAL   DNS          6m2s
reviews.bookinfo.svc.cluster2.global                         [reviews.bookinfo.svc.cluster2.global]                         MESH_INTERNAL   DNS          6m2s

Notice how the output of the following command includes services that are federated from your other cluster, cluster1.

kubectl get serviceentry -n istio-system --context $REMOTE_CONTEXT2

NAME                                                       HOSTS                                                        LOCATION        RESOLUTION   AGE
details.bookinfo.svc.cluster1.global                       [details.bookinfo.svc.cluster1.global]                       MESH_INTERNAL   DNS          2m5s     
istio-ingressgateway.istio-system.svc.cluster1.global      [istio-ingressgateway.istio-system.svc.cluster1.global]      MESH_INTERNAL   DNS          5m18s    
productpage.bookinfo.svc.cluster1.global                   [productpage.bookinfo.svc.cluster1.global]                   MESH_INTERNAL   DNS          55s      
ratings.bookinfo.svc.cluster1.global                       [ratings.bookinfo.svc.cluster1.global]                       MESH_INTERNAL   DNS          7m2s     
reviews.bookinfo.svc.cluster1.global                       [reviews.bookinfo.svc.cluster1.global]                       MESH_INTERNAL   DNS          90s

Next steps

Now, you can route traffic across your clusters with end-to-end mTLS and verify that the certificates use a shared root of trust.

You might try integrating a certificate manager provider, such as AWS or Vault.