Create a root trust policy in Gloo Mesh Enterprise to automatically set up:

  • A managed Istio root certificate authority (CA) with Gloo self-signed certificates in the management cluster.
  • An intermediate CA in each workload cluster that you register. The intermediate CAs are used to automatically sign and issue leaf certificates for the workloads in the service mesh so that they can securely communicate with each other via mutual TLS (mTLS) across cluster boundaries.

For more information about this root trust policy approach and the components that are set up for you, see Option 1: Use Gloo self-signed CA certificates.

Before you begin

  1. Complete the multicluster getting started guide to set up the following testing environment.

    • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
    • The Gloo meshctl CLI, along with other CLI tools such as kubectl and istioctl.
    • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
    • Istio installed in the workload clusters.
    • A simple Gloo workspace setup.
  2. Install Bookinfo and other sample apps.

  3. Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The generated certificate that connects workload clusters to the management cluster uses the context name as a SAN specification, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.

      export MGMT_CLUSTER=<mgmt-cluster-name>
export REMOTE_CLUSTER1=<remote-cluster1-name>
export REMOTE_CLUSTER2=<remote-cluster2-name>
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT1=<remote-cluster1-context>
export REMOTE_CONTEXT2=<remote-cluster2-context>

Step 1: Set up the root trust policy for self-signed root CA certificates

  1. Create a root trust policy in the management cluster. By default, the root and intermediate CA certificates that are created for you are valid for a year. You can customize the number of days a certificate is valid for by specifying the ttlDays in the mgmtServerCA (root CA) and intermediateCertOptions (intermediate CA) section of your root trust policy.

    The following example policy creates a self-signed root CA certificate that is valid for 730 days, and an intermediate CA certificate that is valid for 1 day. To learn more about how Gloo Mesh Enterprise rotates the CA certificates based on these settings, see Certificate rotation overview.

      kubectl apply --context $MGMT_CONTEXT -f- << EOF 
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust-policy
  namespace: gloo-mesh
spec:
  config:
    intermediateCertOptions:
      secretRotationGracePeriodRatio: 0.1
      ttlDays: 1
    mgmtServerCa: 
      generated:
        ttlDays: 730
EOF

  2. Verify that the root-trust-policy.gloo-mesh Kubernetes secret was created in the gloo-mesh namespace on the management cluster. This secret represents the root CA and is used by the certificate issuer service in the Gloo management server to issue intermediate CA certificates to a workload cluster.

      kubectl get secret root-trust-policy.gloo-mesh -n gloo-mesh --context $MGMT_CONTEXT

    Example output:

      NAME                          TYPE                                        DATA   AGE
root-trust-policy.gloo-mesh   admin.gloo.solo.io/generated_signing_cert   4      10s

  3. Verify that the cacerts Kubernetes secret was created in the istio-system namespace on the workload cluster. This secret represents the intermediate CA and is used by the Istio control plane istiod to issue leaf certificates to the workloads in your service mesh.

      kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT1 
kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT2

  4. Verify the certificate chain for the intermediate CA. Because the intermediate CA was derived from the root CA, the root CA must be listed as the root-cert in the cacerts Kubernetes secret.

    1. Get the root CA certificate from the root-trust-policy.gloo-mesh secret on the management cluster.
        kubectl get secret root-trust-policy.gloo-mesh -n gloo-mesh -o jsonpath='{.data.ca-cert\.pem}' --context $MGMT_CONTEXT| base64 --decode
    2. In each workload cluster, get the root CA certificate that is listed as the root-cert in the cacerts Kubernetes secret.
        kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT1 -o jsonpath='{.data.root-cert\.pem}' | base64 --decode
kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT2 -o jsonpath='{.data.root-cert\.pem}' | base64 --decode
    3. Verify that the root CA certificate that is listed in the intermediate CA secret (4.2) matches the root CA certificate that was created by the Gloo root trust policy (4.1).

  5. Restart istiod and the sample apps that you deployed as part of the getting started guide to apply the updated certificates.

      kubectl rollout restart deployment istiod-$REVISION -n istio-system --context $REMOTE_CONTEXT1
kubectl rollout restart deployment istiod-$REVISION -n istio-system --context $REMOTE_CONTEXT2
kubectl rollout restart deployment details-v1 productpage-v1 ratings-v1 reviews-v1 reviews-v2 -n bookinfo --context $REMOTE_CONTEXT1
kubectl rollout restart deployment ratings-v1 reviews-v3 -n bookinfo --context $REMOTE_CONTEXT2
kubectl rollout restart deployment httpbin -n httpbin --context $REMOTE_CONTEXT1
kubectl rollout restart deployment helloworld-v1 helloworld-v2 -n helloworld --context $REMOTE_CONTEXT1
kubectl rollout restart deployment helloworld-v3 helloworld-v4 -n helloworld --context $REMOTE_CONTEXT2

Step 2: Verify workloads can communicate across cluster boundaries

With your root trust policy in place, verify that you can route mTLS traffic within your mesh across cluster boundaries.

  1. Create an Istio policy to enforce mTLS between workloads in the bookinfo namespace in both workload clusters.

      kubectl apply -n bookinfo --context $REMOTE_CONTEXT1 -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
EOF
  
      kubectl apply -n bookinfo --context $REMOTE_CONTEXT2 -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
EOF

  2. Create a virtual destination so that you can route requests from cluster 1 to the reviews app in cluster 2.

      kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata: 
  name: reviews-vd
spec:
  hosts:
  - reviews.vd
  ports:
  - number: 80
    protocol: HTTP
    targetPort:
      name: http
  services:
  - cluster: $REMOTE_CLUSTER1
    labels:
      app: reviews
EOF

  3. Verify that you can send a request from the product page app in cluster 1 to the reviews-v3 app in cluster 2.

      kubectl exec $(kubectl get pod -l app=productpage -A --context ${REMOTE_CONTEXT1}  -o jsonpath='{.items[0].metadata.name}') -n bookinfo -c curl --context ${REMOTE_CONTEXT1} -- curl -vik reviews.vd/reviews/3

    Example output:

      HTTP/1.1 200 OK
< x-powered-by: Servlet/3.1
< content-type: application/json
< date: Thu, 16 Mar 2023 18:21:42 GMT
< content-language: en-US
< content-length: 436
< x-envoy-upstream-service-time: 1455
< server: envoy
< 
{ [436 bytes data]
100   436  100   436    0     0    29x-powered-by: Servlet/3.1
content-type: application/json
date: Thu, 16 Mar 2023 18:21:42 GMT
content-language: en-US
content-length: 436
x-envoy-upstream-service-time: 1455
server: envoy

{"id": "3","podname": "reviews-v3-58b6479b-q2tzn","clustername": "null","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "red"}},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "red"}}]}4      0  0:00:01  0:00:01 --:--:--   294
* Connection #0 to host reviews.vd left intact

  4. Verify that the product page app in cluster 1 and reviews-v3 app in cluster 2 share the same root CA.

    1. List the Istio proxies in your clusters and get the name of the product page (cluster 1) and reviews-v3 (cluster 2) proxies.

        istioctl --context $REMOTE_CONTEXT1 ps
istioctl --context $REMOTE_CONTEXT2 ps

    2. Get the TLS credentials that the product page and reviews-v3 proxies use for mTLS connections.

        istioctl --context $REMOTE_CONTEXT1 pc secret <product-page-proxy-name> -o yaml
istioctl --context $REMOTE_CONTEXT2 pc secret <reviews-v3-proxy-name> -o yaml

    3. In your CLI output, look for the root certificate in the ROOTCA section and copy the inlineBytes content.

      Example output:

            ...
      - lastUpdated: "2022-11-08T09:53:16.743Z"
        name: ROOTCA
        secret:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
          name: ROOTCA
          validationContext:
            trustedCa:
              inlineBytes: <root-certificate>

    4. Decode the content and verify that both proxies list the same root CA certificate.

        echo "<inlineBytes-content>" | base64 -d