Example relay and Istio configuration
Use AWS Certificate Manager (ACM) to create and manage the root and intermediate CA certificates for Gloo Mesh Enterprise.
For an overview of different options to manage certificates, see Certificate architectures.
Example ACM hybrid architecture
The following figure outlines a hybrid configuration to manage certificates in ACM. Both the root CA and relay intermediate CA are stored within ACM. Gloo Mesh continues to issue Istio CA intermediate certificates, but by using an external root CA you can more easily rotate the Gloo Mesh issuing CA certificate at a later time.
Example configuration script
You can use the following script as an example for setting up ACM to manage the root CA and relay intermediate CA for your Gloo Mesh environment. Before you run the script, make sure to update the highlighted lines with the information for your organization.
#!/bin/bash
set -e
# Purpose:
# Creates one Root CA and Subordinate CA/s in AWS Private Certificate Authority(PCA).
# Subordinate CA is meant to be used to setup the CA cert for Istio.
# Note: Please feel free to edit the following section as per your need for the CA Subject details.
export COUNTRY="US"
export ORGANIZATION="Solo.io"
export ORGANIZATIONAL_UNIT="Consulting"
export STATE="MA"
export LOCALITY="Boston"
# Note: Please feel free to edit the values for Validity time of the Root cert and the Subordinate cert.
export ROOT_CERT_VALIDITY_IN_DAYS=3650
export SUBORDINATE_CERT_VALIDITY_IN_DAYS=1825
##
# Process Flow for AWS Private CA setup
# - Create config json file with details for the Certificate Authority.
# - Create CA (Root or Subordinate) in AWS Private CA.
# - Download CSR file corresponding to the newly created Private CA.
# - Issue certificate for the CA with the help of the downloaded CSR and the Private CA ARN.
# > Note: Use "--certificate-authority-arn" parameter for issuing a cert for Subordinate/Intermediate CA)
# - Import this certificate in AWS Private CA.
##
echo
echo "###########################################################"
echo " Creating Root CA"
echo " Generated and managed by ACM"
echo "###########################################################"
cat <<EOF > ca_config_root_ca.json
{
"KeyAlgorithm":"RSA_2048",
"SigningAlgorithm":"SHA256WITHRSA",
"Subject":{
"Country":"${COUNTRY}",
"Organization":"${ORGANIZATION}",
"OrganizationalUnit":"${ORGANIZATIONAL_UNIT}",
"State":"${STATE}",
"Locality":"${LOCALITY}",
"CommonName":"Root CA"
}
}
EOF
echo
echo "[INFO] Creates the root private certificate authority (CA)."
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/create-certificate-authority.html
ROOT_CAARN=$(aws acm-pca create-certificate-authority \
--certificate-authority-configuration file://ca_config_root_ca.json \
--certificate-authority-type "ROOT" \
--idempotency-token 01234567 \
--output json \
--tags Key=Name,Value=RootCA | jq -r '.CertificateAuthorityArn')
echo "[INFO] Sleeping for 15 seconds for CA creation to be completed..."
sleep 15
echo "[DEBUG] ARN of Root CA=${ROOT_CAARN}"
echo "[INFO] download Root CA CSR from AWS"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate-authority-csr.html
aws acm-pca get-certificate-authority-csr \
--certificate-authority-arn "${ROOT_CAARN}" \
--output text > root-ca.csr
echo "[INFO] Issue Root Certificate. Valid for ${ROOT_CERT_VALIDITY_IN_DAYS} days"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/issue-certificate.html
ROOT_CERTARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--csr fileb://root-ca.csr \
--signing-algorithm "SHA256WITHRSA" \
--template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
--validity Value=${ROOT_CERT_VALIDITY_IN_DAYS},Type="DAYS" \
--idempotency-token 1234567 \
--output json | jq -r '.CertificateArn')
echo "[INFO] Sleeping for 15 seconds for cert issuance to be completed..."
sleep 15
echo "[DEBUG] ARN of Root Certificate=${ROOT_CERTARN}"
echo "[INFO] Retrieves root certificate from private CA and save locally as root-ca.pem"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate.html
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${ROOT_CERTARN}" \
--output text > root-ca.pem
echo "[INFO] Import the signed Private CA certificate for the CA specified by the ARN into ACM PCA"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/import-certificate-authority-certificate.html
aws acm-pca import-certificate-authority-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate fileb://root-ca.pem
echo "-----------------------------------------------------------"
echo "ARN of Root CA is ${ROOT_CAARN}"
echo "-----------------------------------------------------------"
##
# Intermediate CA setup section
# Note: If you wish to create more than 1 subordinate CAs please add more constants in the for loop
##
for CA_FOR_COMPONENT in "Istio"
do
echo
echo "###########################################################"
echo " Create Intermediate CA for ${CA_FOR_COMPONENT}"
echo " Generated and managed by ACM, signed by Root CA"
echo "###########################################################"
cat <<EOF > "ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json"
{
"KeyAlgorithm":"RSA_2048",
"SigningAlgorithm":"SHA256WITHRSA",
"Subject":{
"Country":"${COUNTRY}",
"Organization":"${ORGANIZATION}",
"OrganizationalUnit":"${ORGANIZATIONAL_UNIT}",
"State":"${STATE}",
"Locality":"${LOCALITY}",
"CommonName":"Intermediate CA ${CA_FOR_COMPONENT}"
}
}
EOF
echo "[INFO] Create the Subordinate/Intermediate private certificate authority (CA) for ${CA_FOR_COMPONENT}"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/create-certificate-authority.html
SUBORDINATE_CAARN=$(aws acm-pca create-certificate-authority \
--certificate-authority-configuration file://ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json \
--certificate-authority-type "SUBORDINATE" \
--idempotency-token 01234567 \
--tags Key=Name,Value="SubordinateCA-${CA_FOR_COMPONENT}" | jq -r '.CertificateAuthorityArn')
echo "[INFO] Sleeping for 15 seconds for CA creation to be completed..."
sleep 15
echo "[DEBUG] ARN of Subordinate CA for ${CA_FOR_COMPONENT}=${SUBORDINATE_CAARN}"
echo "[INFO] Download Intermediate CA CSR from AWS"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate-authority-csr.html
aws acm-pca get-certificate-authority-csr \
--certificate-authority-arn "${SUBORDINATE_CAARN}" \
--output text > "intermediate_ca_${CA_FOR_COMPONENT}.csr"
echo "[INFO] Issue Intermediate Certificate for ${CA_FOR_COMPONENT}. Valid for ${SUBORDINATE_CERT_VALIDITY_IN_DAYS} days."
SUBORDINATE_CERTARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--csr fileb://intermediate_ca_${CA_FOR_COMPONENT}.csr \
--signing-algorithm "SHA256WITHRSA" \
--template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen1/V1 \
--validity Value=${SUBORDINATE_CERT_VALIDITY_IN_DAYS},Type="DAYS" \
--idempotency-token 1234567 \
--output json | jq -r '.CertificateArn')
echo "[INFO] Sleeping for 15 seconds for cert issuance to be completed..."
sleep 15
echo "[DEBUG] ARN of Subordinate CA Certificate for ${CA_FOR_COMPONENT}=${SUBORDINATE_CERTARN}"
echo "[INFO] Retrieve Intermediate certificate from private CA and save locally as intermediate-cert.pem"
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${SUBORDINATE_CERTARN}" \
--output json | jq -r '.Certificate' > "intermediate-cert-${CA_FOR_COMPONENT}.pem"
echo "[INFO] Retrieve Intermediate certificate chain from private CA and save locally as intermediate-cert-chain.pem"
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${SUBORDINATE_CERTARN}" \
--output json | jq -r '.CertificateChain' > "intermediate-cert-chain-${CA_FOR_COMPONENT}.pem"
echo "[INFO] Import the certificate into ACM PCA"
aws acm-pca import-certificate-authority-certificate \
--certificate-authority-arn "${SUBORDINATE_CAARN}" \
--certificate fileb://intermediate-cert-${CA_FOR_COMPONENT}.pem \
--certificate-chain fileb://intermediate-cert-chain-${CA_FOR_COMPONENT}.pem
echo "-----------------------------------------------------------"
echo "ARN of ${CA_FOR_COMPONENT} CA is ${SUBORDINATE_CAARN}"
echo "-----------------------------------------------------------"
done
```
<h2 id="istio-cert-rotate">Rotating certificates for Istio workloads</h2>
<p>When certificates are issued, pods that are managed by Istio must be restarted to ensure they pick up the new certificates. The certificate issuer creates a PodBounceDirective, which contains the namespaces and labels of the pods that must be restarted. For more information about how certificate rotation works in Istio, review the video series in <a href="https://www.solo.io/blog/videos-avoiding-downtime-with-istio-certificate-rotation/">this blog post</a>.</p>
<p><strong>Note</strong>: To avoid potential downtime for your apps in production, disable the PodBounceDirective feature by setting <code>autoRestartPods</code> to <code>false</code>. Then, control pod restarts in another way, such as a rolling update.</p>
<ol>
<li>
<p>Get your root trust policies.</p>
<div class="highlight"><pre class="chroma"><code class="language-shell" data-lang="shell">kubectl get roottrustpolicy --context <span class="si">${</span><span class="nv">MGMT_CONTEXT</span><span class="si">}</span> -A
</code></pre></div></li>
<li>
<p>In the root trust policy, remove or set the <code>autoRestartPods</code> field to <code>false</code>.</p>
<div class="highlight"><pre class="chroma"><code class="language-shell" data-lang="shell">kubectl edit roottrustpolicy --context <span class="si">${</span><span class="nv">MGMT_CONTEXT</span><span class="si">}</span> -n <namespace> <root-trust-policy>
</code></pre></div><div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml">apiVersion<span class="p">:</span><span class="w"> </span>admin.gloo.solo.io/v2<span class="w">
</span><span class="w"></span>kind<span class="p">:</span><span class="w"> </span>RootTrustPolicy<span class="w">
</span><span class="w"></span>metadata<span class="p">:</span><span class="w">
</span><span class="w"> </span>name<span class="p">:</span><span class="w"> </span>istio-ingressgateway<span class="w">
</span><span class="w"> </span>namespace<span class="p">:</span><span class="w"> </span>gloo-mesh<span class="w">
</span><span class="w"></span>spec<span class="p">:</span><span class="w">
</span><span class="w"> </span>config<span class="p">:</span><span class="w">
</span><span class="w"> </span>autoRestartPods<span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="w"> </span>...<span class="w">
</span></code></pre></div></li>
<li>
<p>To ensure pods pick up the new certificates, restart the <code>istiod</code> pod in each remote cluster.</p>
<div class="highlight"><pre class="chroma"><code class="language-sh" data-lang="sh">kubectl --context <span class="o">{</span><span class="nv">$REMOTE_CONTEXT</span><span class="o">}</span> -n istio-system patch deployment istiod <span class="se">\
</span><span class="se"></span> -p <span class="s2">"{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"</span>
</code></pre></div></li>
<li>
<p>Restart your app pods that are managed by Istio, such as by using a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment">rolling update strategy</a>.</p>
</li>
</ol>