☰
Navigation :

Example relay and Istio configuration

Use AWS Certificate Manager (ACM) to create and manage the root and intermediate CA certificates for Gloo Mesh Enterprise.

For an overview of different options to manage certificates, see Certificate architectures.

Example ACM hybrid architecture

The following figure outlines a hybrid configuration to manage certificates in ACM. Both the root CA and relay intermediate CA are stored within ACM. Gloo Mesh continues to issue Istio CA intermediate certificates, but by using an external root CA you can more easily rotate the Gloo Mesh issuing CA certificate at a later time.

ACM Root and Intermediate Certs

Example configuration script

You can use the following script as an example for setting up ACM to manage the root CA and relay intermediate CA for your Gloo Mesh environment.

###########################################################
# Gloo Mesh Root CA
# Generated and managed by ACM
###########################################################
echo '''
{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "OrganizationalUnit":"Demo",
      "CommonName":"Gloo Mesh Root CA"
   }
}
''' > ca_config.json
CREATE_CA_RESPONSE=$(aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://ca_config.json \
     --certificate-authority-type "ROOT" \
     --idempotency-token 01234567 \
     --tags  Key=Name,Value=Gloo-Mesh-RootCA| jq -r '.CertificateAuthorityArn')
# Response
# {
#   "CertificateAuthorityArn": "arn:aws:acm-pca:us-east-1:123456789:certificate-authority/123456789-debf-4513-89f7-c1834d5ffbd5"
# }

CAARN=$CREATE_CA_RESPONSE

# download Root CA CSR from AWS
aws acm-pca get-certificate-authority-csr \
    --certificate-authority-arn $CAARN \
    --output text > root-ca.csr

# Issue Root Certificate
ISSUE_CERTIFICATE_RESPONSE=$(aws acm-pca issue-certificate \
    --certificate-authority-arn $CAARN \
    --csr fileb://root-ca.csr \
    --signing-algorithm "SHA256WITHRSA" \
    --template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
    --validity Value=3650,Type="DAYS" \
    --idempotency-token 1234567 \
    --output json | jq -r '.CertificateArn')

CERTARN=$ISSUE_CERTIFICATE_RESPONSE

# Download Certificate
aws acm-pca get-certificate \
    --certificate-authority-arn $CAARN \
    --certificate-arn $CERTARN \
    --output text > root-ca.pem

# Upload certificate to AWS
aws acm-pca import-certificate-authority-certificate \
    --certificate-authority-arn $CAARN \
    --certificate fileb://root-ca.pem

###########################################################
# Gloo Mesh Intermediate CA for Istio
# Generated locally, but signed by Gloo Mesh Root CA
###########################################################

# Generate Intermediate CA Certificate Request
CLUSTER_NAME=cluster-1
cat > "gloo-mesh-intermediate.conf" <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = critical,CA:TRUE
keyUsage = digitalSignature, keyEncipherment, keyCertSign
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = $CLUSTER_NAME.gloo-mesh.agent
EOF

openssl genrsa -out gloo-mesh-intermediate.key 2048
openssl req -new -sha256 -key gloo-mesh-intermediate.key -config gloo-mesh-intermediate.conf -out gloo-mesh-intermediate.csr -subj "/CN=gloo-mesh-ca"

# Sign Gloo Mesh Intermediate Certificate
ISSUE_CERTIFICATE_RESPONSE=$(aws acm-pca issue-certificate \
    --certificate-authority-arn $CAARN \
    --csr fileb://gloo-mesh-intermediate.csr \
    --signing-algorithm "SHA256WITHRSA" \
    --template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen1/V1 \
    --validity Value=365,Type="DAYS" \
    --idempotency-token 1234567 \
    --output json | jq -r '.CertificateArn')

INT_CAARN=$ISSUE_CERTIFICATE_RESPONSE

# Download the Gloo Mesh Intermediate Certificate
aws acm-pca get-certificate \
    --certificate-authority-arn $CAARN \
    --certificate-arn $INT_CAARN \
    --output text > gloo-mesh-intermediate.pem

###########################################################
# Relay Intermediate CA
# Generated and managed by ACM, signed by Gloo Mesh Root CA
###########################################################
echo '''
{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "Country":"US",
      "Organization":"Solo.io",
      "OrganizationalUnit":"Demo",
      "State":"MA",
      "Locality":"Boson",
      "CommonName":"Relay Intermediate CA"
   }
}
''' > relay_int_config.json
CREATE_CA_RESPONSE=$(aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://relay_int_config.json \
     --certificate-authority-type "SUBORDINATE" \
     --idempotency-token 01234567 \
     --tags  Key=Name,Value=Gloo-Mesh-RootCA| jq -r '.CertificateAuthorityArn')

REALY_CAARN=$CREATE_CA_RESPONSE

# download Relay CA CSR from AWS
aws acm-pca get-certificate-authority-csr \
    --certificate-authority-arn $REALY_CAARN \
    --output text > relay-int-ca.csr

# Issue Root Certificate
ISSUE_CERTIFICATE_RESPONSE=$(aws acm-pca issue-certificate \
    --certificate-authority-arn $CAARN \
    --csr fileb://relay-int-ca.csr \
    --signing-algorithm "SHA256WITHRSA" \
    --template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen1/V1 \
    --validity Value=1825,Type="DAYS" \
    --idempotency-token 1234567 \
    --output json | jq -r '.CertificateArn')

CERTARN=$ISSUE_CERTIFICATE_RESPONSE

# Download Certificate
aws acm-pca get-certificate \
    --certificate-authority-arn $CAARN \
    --certificate-arn $CERTARN \
    --output json | jq -r '.Certificate' > relay-int.pem
aws acm-pca get-certificate \
    --certificate-authority-arn $CAARN \
    --certificate-arn $CERTARN \
    --output json | jq -r '.CertificateChain' > relay-int-chain.pem

# Upload certificate to AWS
aws acm-pca import-certificate-authority-certificate \
    --certificate-authority-arn $REALY_CAARN \
    --certificate fileb://relay-int.pem \
    --certificate-chain fileb://relay-int-chain.pem

###########################################################
# Relay (gloo-mesh-mgmt-server) mTLS Server Certificate
# Generated by the Relay Intermediate CA in ACM
###########################################################

RELAY_SERVER_CERT_NAME=relay-server-tls
# server cert
cat > "${RELAY_SERVER_CERT_NAME}.conf" <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh
EOF

openssl genrsa -out "${RELAY_SERVER_CERT_NAME}.key" 2048
openssl req -new -key "${RELAY_SERVER_CERT_NAME}.key" -out ${RELAY_SERVER_CERT_NAME}.csr -subj "/CN=gloo-mesh-mgmt-server-ca" -config "${RELAY_SERVER_CERT_NAME}.conf"

# Sign Relay Server Certificate
ISSUE_CERTIFICATE_RESPONSE=$(aws acm-pca issue-certificate \
    --certificate-authority-arn $REALY_CAARN \
    --csr fileb://${RELAY_SERVER_CERT_NAME}.csr \
    --signing-algorithm "SHA256WITHRSA" \
    --validity Value=365,Type="DAYS" \
    --idempotency-token 1234567 \
    --output json | jq -r '.CertificateArn')

RELAY_CERT_ARN=$ISSUE_CERTIFICATE_RESPONSE

# Download the Gloo Mesh Intermediate Certificate
aws acm-pca get-certificate \
    --certificate-authority-arn $REALY_CAARN \
    --certificate-arn $RELAY_CERT_ARN \
    --output text > ${RELAY_SERVER_CERT_NAME}.pem


###########################################################
# Relay (gloo-mesh-agent) mTLS Client Certificate
# Generated by the Relay Intermediate CA in ACM
# One certificate must be generated per cluster
###########################################################
RELAY_CLIENT_CERT_NAME=relay-server-tls
CLUSTER_NAME=cluster-1
# server client
cat > "${RELAY_CLIENT_CERT_NAME}.conf" <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
subjectAltName = @alt_names
[alt_names]
DNS = *.gloo-mesh
EOF

openssl genrsa -out "${RELAY_CLIENT_CERT_NAME}.key" 2048
openssl req -new -key "${RELAY_CLIENT_CERT_NAME}.key" -out ${RELAY_CLIENT_CERT_NAME}.csr -subj "/CN=$CLUSTER_NAME" -config "${RELAY_CLIENT_CERT_NAME}.conf"

# Sign Relay Server Certificate
ISSUE_CERTIFICATE_RESPONSE=$(aws acm-pca issue-certificate \
    --certificate-authority-arn $REALY_CAARN \
    --csr fileb://${RELAY_CLIENT_CERT_NAME}.csr \
    --signing-algorithm "SHA256WITHRSA" \
    --validity Value=365,Type="DAYS" \
    --idempotency-token 1234567 \
    --output json | jq -r '.CertificateArn')

RELAY_CERT_ARN=$ISSUE_CERTIFICATE_RESPONSE

# Download the Gloo Mesh Intermediate Certificate
aws acm-pca get-certificate \
    --certificate-authority-arn $REALY_CAARN \
    --certificate-arn $RELAY_CERT_ARN \
    --output text > ${RELAY_CLIENT_CERT_NAME}.pem

Rotating certificates for Istio workloads

When certificates are issued, pods that are managed by Istio must be restarted to ensure they pick up the new certificates. The certificate issuer creates a PodBounceDirective, which contains the namespaces and labels of the pods that must be restarted. For more information about how certificate rotation works in Istio, review the video series in this blog post.

Note: To avoid potential downtime for your apps in production, disable the PodBounceDirective feature by setting autoRestartPods to false. Then, control pod restarts in another way, such as a rolling update.

  1. Get your root trust policies.

    kubectl get roottrustpolicy --context ${MGMT_CONTEXT} -A

  2. In the root trust policy, remove or set the autoRestartPods field to false.

    kubectl edit roottrustpolicy --context ${MGMT_CONTEXT} -n <namespace> <root-trust-policy>

    apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    autoRestartPods: false
    ...

  3. To ensure pods pick up the new certificates, restart the istiod pod in each remote cluster.

    kubectl --context {$REMOTE_CONTEXT} -n istio-system patch deployment istiod \
    -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"

  4. Restart your app pods that are managed by Istio, such as by using a rolling update strategy.