Manage the entire Istio CA lifecycle
Use a root trust policy to set up a root certificate authority (CA) in the management cluster and an intermediate CA in each workload cluster.
Create a root trust policy in Gloo Mesh Enterprise to automatically set up:
- A managed Istio root certificate authority (CA) with Gloo self-signed certificates in the management cluster.
- An intermediate CA in each workload cluster that you register. The intermediate CAs are used to automatically sign and issue leaf certificates for the workloads in the service mesh so that they can securely communicate with each other via mutual TLS (mTLS) across cluster boundaries.
For more information about this root trust policy approach and the components that are set up for you, see Option 1: Use Gloo self-signed CA certificates.
Before you begin
This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.
Complete the multicluster getting started guide to set up the following testing environment.
- Three clusters along with environment variables for the clusters and their Kubernetes contexts.
- The Gloo
meshctl
CLI, along with other CLI tools such askubectl
andistioctl
. - The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
- Istio installed in the workload clusters.
- A simple Gloo workspace setup.
- Install Bookinfo and other sample apps.
Save the kubeconfig contexts for your clusters. Run
kubectl config get-contexts
, look for your cluster in theCLUSTER
column, and get the context name in theNAME
column. Note: Do not use context names with underscores. The generated certificate that connects workload clusters to the management cluster uses the context name as a SAN specification, and underscores in SAN are not FQDN compliant. You can rename a context by runningkubectl config rename-context "<oldcontext>" <newcontext>
.export MGMT_CLUSTER=<mgmt-cluster-name> export REMOTE_CLUSTER1=<remote-cluster1-name> export REMOTE_CLUSTER2=<remote-cluster2-name> export MGMT_CONTEXT=<management-cluster-context> export REMOTE_CONTEXT1=<remote-cluster1-context> export REMOTE_CONTEXT2=<remote-cluster2-context>
Step 1: Set up the root trust policy for self-signed root CA certificates
Create a root trust policy in the management cluster. By default, the root and intermediate CA certificates that are created for you are valid for a year. You can customize the number of days a certificate is valid for by specifying the
ttlDays
in themgmtServerCA
(root CA) andintermediateCertOptions
(intermediate CA) section of your root trust policy.The following example policy creates a self-signed root CA certificate that is valid for 730 days, and an intermediate CA certificate that is valid for 1 day. To learn more about how Gloo Mesh Enterprise rotates the CA certificates based on these settings, see Certificate rotation overview.
kubectl apply --context $MGMT_CONTEXT -f- << EOF apiVersion: admin.gloo.solo.io/v2 kind: RootTrustPolicy metadata: name: root-trust-policy namespace: gloo-mesh spec: config: intermediateCertOptions: secretRotationGracePeriodRatio: 0.1 ttlDays: 1 mgmtServerCa: generated: ttlDays: 730 EOF
Verify that the
root-trust-policy.gloo-mesh
Kubernetes secret was created in thegloo-mesh
namespace on the management cluster. This secret represents the root CA and is used by the certificate issuer service in the Gloo management server to issue intermediate CA certificates to a workload cluster.kubectl get secret root-trust-policy.gloo-mesh -n gloo-mesh --context $MGMT_CONTEXT
Example output:
NAME TYPE DATA AGE root-trust-policy.gloo-mesh admin.gloo.solo.io/generated_signing_cert 4 10s
Verify that the
cacerts
Kubernetes secret was created in theistio-system
namespace on the workload cluster. This secret represents the intermediate CA and is used by the Istio control plane istiod to issue leaf certificates to the workloads in your service mesh.kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT1 kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT2
Verify the certificate chain for the intermediate CA. Because the intermediate CA was derived from the root CA, the root CA must be listed as the
root-cert
in thecacerts
Kubernetes secret.- Get the root CA certificate from the
root-trust-policy.gloo-mesh
secret on the management cluster.kubectl get secret root-trust-policy.gloo-mesh -n gloo-mesh -o jsonpath='{.data.ca-cert\.pem}' --context $MGMT_CONTEXT| base64 --decode
- In each workload cluster, get the root CA certificate that is listed as the
root-cert
in thecacerts
Kubernetes secret.kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT1 -o jsonpath='{.data.root-cert\.pem}' | base64 --decode kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT2 -o jsonpath='{.data.root-cert\.pem}' | base64 --decode
- Verify that the root CA certificate that is listed in the intermediate CA secret (4.2) matches the root CA certificate that was created by the Gloo root trust policy (4.1).
- Get the root CA certificate from the
Step 2: Verify workloads can communicate across cluster boundaries
With your root trust policy in place, verify that you can route mTLS traffic within your mesh across cluster boundaries.
Create an Istio policy to enforce mTLS between workloads in the bookinfo namespace in both workload clusters.
Create a virtual destination so that you can route requests from cluster 1 to the reviews app in cluster 2.
kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF apiVersion: networking.gloo.solo.io/v2 kind: VirtualDestination metadata: name: reviews-vd spec: hosts: - reviews.vd ports: - number: 80 protocol: HTTP targetPort: name: http services: - cluster: $REMOTE_CLUSTER1 labels: app: reviews EOF
Verify that you can send a request from the product page app in cluster 1 to the reviews-v3 app in cluster 2.
kubectl exec $(kubectl get pod -l app=productpage -A --context ${REMOTE_CONTEXT1} -o jsonpath='{.items[0].metadata.name}') -n bookinfo -c curl --context ${REMOTE_CONTEXT1} -- curl -vik reviews.vd/reviews/3
Example output:
HTTP/1.1 200 OK < x-powered-by: Servlet/3.1 < content-type: application/json < date: Thu, 16 Mar 2023 18:21:42 GMT < content-language: en-US < content-length: 436 < x-envoy-upstream-service-time: 1455 < server: envoy < { [436 bytes data] 100 436 100 436 0 0 29x-powered-by: Servlet/3.1 content-type: application/json date: Thu, 16 Mar 2023 18:21:42 GMT content-language: en-US content-length: 436 x-envoy-upstream-service-time: 1455 server: envoy {"id": "3","podname": "reviews-v3-58b6479b-q2tzn","clustername": "null","reviews": [{ "reviewer": "Reviewer1", "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "red"}},{ "reviewer": "Reviewer2", "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "red"}}]}4 0 0:00:01 0:00:01 --:--:-- 294 * Connection #0 to host reviews.vd left intact
Verify that the product page app in cluster 1 and reviews-v3 app in cluster 2 share the same root CA.
List the Istio proxies in your clusters and get the name of the product page (cluster 1) and reviews-v3 (cluster 2) proxies.
istioctl --context $REMOTE_CONTEXT1 ps istioctl --context $REMOTE_CONTEXT2 ps
Get the TLS credentials that the product page and reviews-v3 proxies use for mTLS connections.
istioctl --context $REMOTE_CONTEXT1 pc secret <product-page-proxy-name> -o yaml istioctl --context $REMOTE_CONTEXT2 pc secret <reviews-v3-proxy-name> -o yaml
In your CLI output, look for the root certificate in the
ROOTCA
section and copy theinlineBytes
content.Example output:
... - lastUpdated: "2022-11-08T09:53:16.743Z" name: ROOTCA secret: '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret name: ROOTCA validationContext: trustedCa: inlineBytes: <root-certificate>
Decode the content and verify that both proxies list the same root CA certificate.
echo "<inlineBytes-content>" | base64 -d