On this page

Manage Istio intermediate CAs

Generate your own Istio root CA certificate, and use the certificate secret to sign and issue intermediate and leaf certificates for the workloads in your service mesh.

Overview

Instead of using Gloo self-signed certificates for the root CA certificate, you can generate your own Istio root CA certificate and key with the certificate management tool of your choice. You then have the following options:

Store the root CA certificate and key in a Kubernetes secret on the management cluster as described in Option 2: Managed intermediate CA.
Store the root CA credentials with your PKI provider for enhanced security. If you decide on this approach, you must create an additional intermediate CA layer as described in Option 3: Managed multi-intermediate CAs and store the intermediate CA credentials in the management cluster. You can then use Gloo Mesh Enterprise to automatically sign and issue intermediate CA certificates for each workload cluster.

warning

Storing the root CA certificate and private key in the Gloo management cluster is not a recommended practice for production. The root CA certificate and key are very sensitive credentials, and if compromised, can be used to issue certificates for all components in your service mesh. To protect the root CA certificate and key, it is recommended to store them in an external certificate management system. For more information, see the managed CA lifecycle Setup options and your options to Bring your own CAs.

Before you begin

info

This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.

Complete the multicluster getting started guide to set up the following testing environment.
- Three clusters along with environment variables for the clusters and their Kubernetes contexts.
- The Gloo meshctl CLI, along with other CLI tools such as kubectl and istioctl.
- The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
- Istio installed in the workload clusters.
- A simple Gloo workspace setup.
Install Bookinfo and other sample apps.
Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The generated certificate that connects workload clusters to the management cluster uses the context name as a SAN specification, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
```
  export MGMT_CLUSTER=<mgmt-cluster-name>
export REMOTE_CLUSTER1=<remote-cluster1-name>
export REMOTE_CLUSTER2=<remote-cluster2-name>
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT1=<remote-cluster1-context>
export REMOTE_CONTEXT2=<remote-cluster2-context>
  
```

Step 1: Create your own root CA certificate

To generate and store your own CA certificate and key, you have two general options:

PKI provider (production environments): Generate the root CA certificate and key with your preferred PKI provider, such as Vault, Google Cloud CA, or AWS Private CA. Then, continue to Step 2: Create a root trust policy.
Generate your own (POC or staging environments): If you do not have a PKI provider, you can use the following tools to generate the certificate and key for the CA.
- Istio certificate generator
- OpenSSL

Istio certificate generator

Istio provides a certificate generator tool that you can use to quickly generate self-signed root and intermediate CA certificates and keys.

Download Istio to your local machine. Make sure to use a supported version that matches the version you plan to run in the workload clusters.
```
  ISTIO_VERSION=1.20.8-patch1
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$ISTIO_VERSION sh -
  
```

Go to the certs directory.

  cd istio-$ISTIO_VERSION/tools/certs

Generate a self-signed root CA certificate and private key.

  make -f Makefile.selfsigned.mk \
ROOTCA_CN="Solo Root CA" \
ROOTCA_ORG=Istio \
root-ca

Store the root CA certificate, private key and certificate chain in a Kubernetes secret on the management cluster.

  cp root-cert.pem ca-cert.pem
cp root-key.pem ca-key.pem
cp root-cert.pem cert-chain.pem

kubectl --context $MGMT_CONTEXT -n gloo-mesh create secret generic my-root-trust-policy.gloo-mesh \
--from-file=./ca-cert.pem \
--from-file=./ca-key.pem \
--from-file=./cert-chain.pem \
--from-file=./root-cert.pem

Continue with Step 2: Create a root trust policy.

OpenSSL

You can use OpenSSL to generate your own root CA certificate and private key.

info
The default openssl version that is included in macOS is LibreSSL, which does not work with these instructions. Make sure that you have the OpenSSL version of openssl, not LibreSSL.
The openssl version must be at least 1.1.
1. Check your openssl version. If you see LibreSSL in the output, continue to the next step.
```
  openssl version
  
```
2. Install the OpenSSL version (not LibreSSL). For example, you might use Homebrew.
```
  brew install openssl
  
```
3. Review the output of the OpenSSL installation for the path of the binary file. You can choose to export the binary to your path, or call the entire path whenever the following steps use an openssl command.
  - For example, openssl might be installed along the following path: /usr/local/opt/openssl@3/bin/
  - To run commands, you can append the path so that your terminal uses this installed version of OpenSSL, and not the default LibreSSL. /usr/local/opt/openssl@3/bin/openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650...
1. Check the openssl version that is installed. If you see LibreSSL in the output, continue to the next step.
```
  openssl version
  
```
2. Install the OpenSSL version (not LibreSSL). For example, you might use Homebrew.
```
  brew install openssl
  
```
3. Review the output of the OpenSSL installation for the path of the binary file. You can choose to export the binary to your path, or call the entire path whenever the following steps use an openssl command.
  - For example, openssl might be installed along the following path: /usr/local/opt/openssl@3/bin/
  - To run commands, you can append the path so that your terminal uses this installed version of OpenSSL, and not the default LibreSSL.
```
  /usr/local/opt/openssl@3/bin/openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650...
  
```

Create a configuration file for the root certificate authority.

  cat > "root-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
[ req_dn ]
O = Istio
CN = Root CA
EOF

Use the root CA configuration file to create a root CA certificate and private key.

  openssl req -new -config root-ca.conf -nodes -out root-ca.crt -keyout root-ca.key

Use the root CA certificate and private key to sign the root CA certificate.

  openssl x509 -in root-ca.crt -out cert.pem -req -signkey root-ca.key -days 360

Verify your root CA certificate settings.

  openssl x509 -in cert.pem -text -noout

Store the root CA certificate, private key and certificate chain in a Kubernetes secret on the management cluster.

  cp cert.pem ca-cert.pem
cp root-ca.key ca-key.pem
cp root-ca.crt cert-chain.pem

kubectl --context $MGMT_CONTEXT -n gloo-mesh create secret generic my-root-trust-policy.gloo-mesh \
--from-file=./ca-cert.pem \
--from-file=./ca-key.pem \
--from-file=./cert-chain.pem \
--from-file=./root-cert.pem

Continue with Step 2: Create a root trust policy.

Step 2: Create a root trust policy

Use a Gloo root trust policy to configure the details of the root and intermediate CA certificates across clusters in the service mesh.

Create the root trust policy in the management cluster and reference the root CA Kubernetes secret that you created in step 1 in the spec.config.mgmtServerCa.secretRef section. You can optionally customize the number of days the root and derived intermediate CA certificates are valid for by specifying the ttlDays in the mgmtServerCA (root CA) and intermediateCertOptions (intermediate CA) of your root trust policy.
The following example policy sets up the root CA with the credentials that you stored in the my-root-trust-policy.gloo-mesh secret. The intermediate CA certificates that are automatically created by Gloo are valid for 1 day. For more information, see Certificate rotation overview.
```
  kubectl apply --context $MGMT_CONTEXT -f- << EOF 
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust-policy
  namespace: gloo-mesh
spec:
  config:
    intermediateCertOptions:
      secretRotationGracePeriodRatio: 0.1
      ttlDays: 1
    mgmtServerCa: 
      secretRef:
         name: my-root-trust-policy.gloo-mesh
         namespace: gloo-mesh
EOF
  
```
Verify that the cacerts Kubernetes secret was created in the istio-system namespace on the workload cluster. This secret represents the intermediate CA and is used by istiod to issue leaf certificates to the workloads in your service mesh.
```
  kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT1 
kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT2
  
```
Verify the certificate chain for the intermediate CA. Because the intermediate CA was derived from the root CA, the root CA must be listed as the root-cert in the cacerts Kubernetes secret.
1. Get the root CA certificate from the root-trust-policy.gloo-mesh secret on the management cluster.
```
  kubectl get secret my-root-trust-policy.gloo-mesh -n gloo-mesh -o jsonpath='{.data.ca-cert\.pem}' --context $MGMT_CONTEXT| base64 --decode
  
```
2. In each workload cluster, get the root CA certificate that is listed as the root-cert in the cacerts Kubernetes secret.
```
  kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT1 -o jsonpath='{.data.root-cert\.pem}' | base64 --decode
kubectl get secret cacerts -n istio-system --context $REMOTE_CONTEXT2 -o jsonpath='{.data.root-cert\.pem}' | base64 --decode
  
```
3. Verify that the root CA certificate that is listed in the intermediate CA secret matches the root CA certificate that was created by the Gloo root trust policy.

Restart istiod and the sample apps that you deployed as part of the getting started guide to apply the updated certificates.

  kubectl rollout restart deployment istiod-$REVISION -n istio-system --context $REMOTE_CONTEXT1
kubectl rollout restart deployment istiod-$REVISION -n istio-system --context $REMOTE_CONTEXT2
kubectl rollout restart deployment details-v1 productpage-v1 ratings-v1 reviews-v1 reviews-v2 -n bookinfo --context $REMOTE_CONTEXT1
kubectl rollout restart deployment ratings-v1 reviews-v3 -n bookinfo --context $REMOTE_CONTEXT2
kubectl rollout restart deployment httpbin -n httpbin --context $REMOTE_CONTEXT1
kubectl rollout restart deployment helloworld-v1 helloworld-v2 -n helloworld --context $REMOTE_CONTEXT1
kubectl rollout restart deployment helloworld-v3 helloworld-v4 -n helloworld --context $REMOTE_CONTEXT2

Step 3: Verify workloads can communicate across cluster boundaries

With your root trust policy in place, verify that you can route mTLS traffic within your mesh across cluster boundaries.

Create an Istio policy to enforce mTLS between workloads in the bookinfo namespace in both workload clusters.

  kubectl apply -n bookinfo --context $REMOTE_CONTEXT1 -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
EOF

  kubectl apply -n bookinfo --context $REMOTE_CONTEXT2 -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
EOF

Create a virtual destination so that you can route requests from cluster 1 to the reviews app in cluster 2.

  kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata: 
  name: reviews-vd
spec:
  hosts:
  - reviews.vd
  ports:
  - number: 80
    protocol: HTTP
    targetPort:
      name: http
  services:
  - cluster: $REMOTE_CLUSTER1
    labels:
      app: reviews
EOF

Verify that you can send a request from the product page app in cluster 1 to the reviews-v3 app in cluster 2.

  kubectl exec $(kubectl get pod -l app=productpage -A --context ${REMOTE_CONTEXT1}  -o jsonpath='{.items[0].metadata.name}') -n bookinfo -c curl --context ${REMOTE_CONTEXT1} -- curl -vik reviews.vd/reviews/3

Example output:

  HTTP/1.1 200 OK
< x-powered-by: Servlet/3.1
< content-type: application/json
< date: Thu, 16 Mar 2023 18:21:42 GMT
< content-language: en-US
< content-length: 436
< x-envoy-upstream-service-time: 1455
< server: envoy
< 
{ [436 bytes data]
100   436  100   436    0     0    29x-powered-by: Servlet/3.1
content-type: application/json
date: Thu, 16 Mar 2023 18:21:42 GMT
content-language: en-US
content-length: 436
x-envoy-upstream-service-time: 1455
server: envoy

{"id": "3","podname": "reviews-v3-58b6479b-q2tzn","clustername": "null","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "red"}},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "red"}}]}4      0  0:00:01  0:00:01 --:--:--   294
* Connection #0 to host reviews.vd left intact

Verify that the product page app in cluster 1 and reviews-v3 app in cluster 2 share the same root CA.

List the Istio proxies in your clusters and get the name of the product page (cluster 1) and reviews-v3 (cluster 2) proxies.
```
  istioctl --context $REMOTE_CONTEXT1 ps
istioctl --context $REMOTE_CONTEXT2 ps
  
```

Get the TLS credentials that the product page and reviews-v3 proxies use for mTLS connections.

  istioctl --context $REMOTE_CONTEXT1 pc secret <product-page-proxy-name> -o yaml
istioctl --context $REMOTE_CONTEXT2 pc secret <reviews-v3-proxy-name> -o yaml

In your CLI output, look for the root certificate in the ROOTCA section and copy the inlineBytes content.

Example output:

      ...
      - lastUpdated: "2022-11-08T09:53:16.743Z"
        name: ROOTCA
        secret:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
          name: ROOTCA
          validationContext:
            trustedCa:
              inlineBytes: <root-certificate>

Decode the content and verify that both proxies list the same root CA certificate.
```
  echo "<inlineBytes-content>" | base64 -d
  
```

Manage Istio intermediate CAs

Overview link

Before you begin link

Step 1: Create your own root CA certificate link

Istio certificate generator link

OpenSSL link

Step 2: Create a root trust policy link

Step 3: Verify workloads can communicate across cluster boundaries link