Navigation :

Integrating Gloo Edge and Let's Encrypt with cert-manager

This document shows how to secure your traffic using Gloo Edge, Let’s Encrypt, and cert-manager. This guide assumes you already have a Kubernetes cluster up and running. Further, it assumes your cluster has a load-balancer such that when Gloo Edge is installed, the proxy service is granted an external IP. This guide will show examples for both the DNS-01 and HTTP-01 challenges.

Prerequisites
DNS-01 Challenge
HTTP-01 Challenge

Prerequisites

Install Gloo Edge

To install Gloo Edge, run:

glooctl install gateway

Install cert manager

The official installation guide is here. You can install with static manifests or helm. For this example we will use the short version - static manifests:

kubectl create namespace cert-manager
kubectl apply --validate=false -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

Utilizing the ACME DNS-01 Challenge

Start by allowing cert manager to configure DNS records in AWS.

Choose between the following options:

Development and testing environments: Use an AWS key pair.
Production environments: Use IAM roles for service accounts (IRSA).

For more details on the access requirements for cert-manager, especially for cross-account cases that are not covered in this guide, see the cert manager docs.

In this example we used the domain name test-123456789.solo.io. We’ll create an A record that maps to the IP address of the gateway proxy that we installed with Gloo Edge.

You can run these commands to update AWS route53 through the AWS command line tool (remember to replace HOSTED_ZONE and RECORD with your values):

export GLOO_HOST=$(kubectl get svc -l gloo=gateway-proxy -n gloo-system -o 'jsonpath={.items[0].status.loadBalancer.ingress[0].ip}')
export RECORD=test-123456789
export HOSTED_ZONE=solo.io.
export ROUTE53_ZONE_ID=$(aws route53 list-hosted-zones|jq -r '.HostedZones[]|select(.Name == "'"$HOSTED_ZONE"'").Id')
export RS='{ "Changes": [{"Action": "UPSERT", "ResourceRecordSet":{"ResourceRecords":[{"Value": "'$GLOO_HOST'"}],"Type": "A","Name": "'$RECORD.$HOSTED_ZONE'","TTL": 300} } ]}'
aws route53 change-resource-record-sets --hosted-zone-id $ROUTE53_ZONE_ID --change-batch "$RS"

Using an AWS key pair

Provide AWS account details to cert-manager

Allow cert-manager access to configure DNS records in AWS. See cert-manager docs for more details on the access requirements for cert-manager.

Once you have configured access, add the access keys as a Kubernetes secret, so that cert-manager can access them:

export ACCESS_KEY_ID=...
export SECRET_ACCESS_KEY=...
kubectl create secret generic aws-creds -n cert-manager --from-literal=access_key_id=$ACCESS_KEY_ID --from-literal=secret_access_key=$SECRET_ACCESS_KEY

Create a cluster issuer

Create a cluster issuer for Let’s Encrypt with Route 53.

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: gloo-system
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: yuval@solo.io
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - dns01:
        route53:
          region: us-east-1
          accessKeyID: $(kubectl -n cert-manager get secret aws-creds -o=jsonpath='{.data.access_key_id}'|base64 --decode)
          secretAccessKeySecretRef:
            name: aws-creds
            key: secret_access_key
EOF

Using AWS IRSA

For production-level setups, use IAM roles for service accounts (IRSA) to give cert manager the necessary access.

Before you begin: Make sure that IRSA is enabled in your EKS cluster. For more information, see the AWS IRSA documentation.

Create role and policy

For convenience, save the following information in environment variables:

export AWS_ACCOUNT=$(aws sts get-caller-identity --query Account | tr -d '"')
export EKS_CLUSTER_NAME=my-eks-cluster-name
export EKS_REGION=us-east-1
export HOSTED_ZONE=solo.io.
export ROUTE53_ZONE_ID=$(aws route53 list-hosted-zones|jq -r '.HostedZones[]|select(.Name == "'"$HOSTED_ZONE"'").Id')
export EKS_HASH=$(aws eks describe-cluster --name ${EKS_CLUSTER_NAME} --query cluster.identity.oidc.issuer | cut -d '/' -f5 | tr -d '"')

Create a policy with the minimum-required access.

cat <<EOF > policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
    --policy-name AwsCertManagerToRoute53 \
    --policy-document file://policy.json

And attach this policy to a role.

cat <<EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT}:oidc-provider/oidc.eks.${EKS_REGION}.amazonaws.com/id/${EKS_HASH}"
      },
      "Condition": {
        "StringEquals": {
          "oidc.eks.${EKS_REGION}.amazonaws.com/id/${EKS_HASH}:sub": "system:serviceaccount:cert-manager:cert-manager"
        }
      }
    }
  ]
}
EOF

aws iam create-role --role-name EksCertManagerRole --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::${AWS_ACCOUNT}:policy/AwsCertManagerToRoute53 --role-name EksCertManagerRole

export IAM_ROLE_ARN=$(aws iam get-role --role-name EksCertManagerRole --query Role.Arn | tr -d '"')

Annotate the cert-manager service account to use this role to manage route53 records.

kubectl annotate sa -n cert-manager cert-manager "eks.amazonaws.com/role-arn"="${IAM_ROLE_ARN}"

To enable the cert-manager deployment to read the ServiceAccount token, modify the cert-manager deployment to define new file system permissions with the following command. You can also make these changes by upgrading the Helm chart that you used to deployed cert-manager, which persists the changes across upgrades.

kubectl patch deployment -n cert-manager cert-manager --type "json" -p '[{"op":"add","path":"/spec/template/spec/securityContext/fsGroup","value":1001}]

Create a cluster issuer

Finally, create a cluster issuer for Let’s Encrypt with Route 53.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: aurelien.tison@solo.io
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - selector:
        dnsZones:
          - "${HOSTED_ZONE}"
      dns01:
        route53:
          region: ${EKS_REGION}
EOF

Check the cluster issuer state

Once account and rights have been defined, wait until the cluster issuer is in ready state:

kubectl get clusterissuer letsencrypt-staging -o jsonpath='{.status.conditions[0].type}{"\n"}'
Ready

Create a certificate for our service

Create the certificate for the Gloo Edge ingress:

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-123456789.solo.io
  namespace: gloo-system
spec:
  secretName: test-123456789.solo.io
  dnsNames:
  - test-123456789.solo.io
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
EOF

Wait a bit and you will see the secret created:

kubectl -n gloo-system  get secret
NAME                   TYPE                                  DATA      AGE
test-123456789.solo.io kubernetes.io/tls                     2         3h
...

Now just create a virtual host with the same secret ref as the name.

Expose the service securly via Gloo Edge

Configure Gloo Edge’s default Virtual Service to route to the function and use the certificates.

cat <<EOF | kubectl create -f -
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: petclinic-ssl
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - test-123456789.solo.io
    routes:
    - matchers:
       - prefix: /
      routeAction:
        single:
          upstream:
              name: default-petclinic-80
              namespace: gloo-system
  sslConfig:
    secretRef:
      name: test-123456789.solo.io
      namespace: gloo-system
EOF

Test!

Now we can open the petclinic application at https://test-123456789.solo.io/.

Utilizing the ACME HTTP-01 Challenge

We just explored how to utilize cert-manager to solve the DNS-01 ACME challenge. While that works great, sometimes a “lighter-weight” solution is desirable. For these situations, the HTTP-01 ACME challenge is a good fit.

We will now illustrate solving the HTTP-01 ACME challenge with Gloo Edge and cert-manager. The HTTP-01 challenge specifically involves the ACME server (Let’s Encrypt) passing a token to your ACME client (cert-manager). The expectation is for that token to be reachable on your domain at a “well known” path, specifically http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>

For this example, we will be using an externally accessible IP (provided through a LoadBalancer Service in a cloud environment) in conjunction with a nip.io domain name. nip.io is a helpful service which allows us to map an arbitrary IP address to a specific domain name via DNS.

These steps are specific for Gloo Edge running in gateway mode. When running in ingress mode, since cert-manager will automatically create Ingress resources, you will not need to add/modify VirtualService resources.

Create an Issuer

First, create a ClusterIssuer which will utilize the http01 solver:

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-http01
  namespace: gloo-system
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: law@solo.io
    privateKeySecretRef:
      name: letsencrypt-staging-http01
    solvers:
    - http01:
        ingress:
          serviceType: ClusterIP
      selector:
        dnsNames:
        - $(glooctl proxy address | cut -f 1 -d ':').nip.io
EOF

Notice the use of the http01 solver. By default, cert-manager will create a Service of type NodePort to be routed via an Ingress. However, since we are running Gloo Edge in gateway mode, incoming traffic is routed via a VirtualService and does not require a NodePort, so we are explicitly setting the serviceType to ClusterIP.

Additionally, we are specifying the dnsName to be a nip.io subdomain with the IP of our external facing LoadBalancer IP. The inline command uses glooctl proxy address to get the external facing IP address of our proxy and we append the ‘nip.io’ domain, leaving us with a domain that looks something like: 34.71.xx.xx.nip.io.

Create the Certificate

Next we will create the actual Certificate which will utilize the ClusterIssuer we just created:

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nip-io
  namespace: default
spec:
  secretName: nip-io-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging-http01
  commonName: $(glooctl proxy address | cut -f 1 -d ':').nip.io
  dnsNames:
    - $(glooctl proxy address | cut -f 1 -d ':').nip.io
EOF

Once this Certificate resource is created, behind the scenes cert-manager will create the relevant CertificateRequest and Order resources. To satisfy this ‘order’, cert-manager will spin up a pod and service that will present the correct token.

Routing to the cert-manager pod

Now that the pod which will serve this token is created, we need to configure Gloo Edge to route to it. In this case, we will create a Virtual Service for our custom domain that will route requests for the path /.well-known/acme-challenge/<TOKEN> to the cert-manager created pod.

We can see this pod present in our default namespace:

% kubectl get pod
NAME                        READY   STATUS    RESTARTS   AGE
cm-acme-http-solver-s69mw   1/1     Running   0          1m6s

And the Service that corresponds to it:

% kubectl get service
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                               AGE
cm-acme-http-solver-f6mdb   ClusterIP      10.35.254.161   <none>           8089/TCP                              2m5s

With Upstream discovery enabled, an Upstream to this Service will automatically be generated:

% glooctl get us default-cm-acme-http-solver-f6mdb-8089
+----------------------------------------+------------+----------+--------------------------------+
|                UPSTREAM                |    TYPE    |  STATUS  |            DETAILS             |
+----------------------------------------+------------+----------+--------------------------------+
| default-cm-acme-http-solver-f6mdb-8089 | Kubernetes | Accepted | svc name:                      |
|                                        |            |          | cm-acme-http-solver-f6mdb      |
|                                        |            |          | svc namespace: default         |
|                                        |            |          | port:          8089            |
|                                        |            |          |                                |
+----------------------------------------+------------+----------+--------------------------------+

In order to view the token value for this Order, we can inspect the Order itself:

kubectl get orders.acme.cert-manager.io nip-io-556035424-1317610542 -o=jsonpath='{.status.authorizations[0].challenges[?(@.type=="http-01")].token}'
q5x9q1C4pPg1RtDEiXK9aMAb9ExpepU4Pp14pGKDPXo

Now we have all the information necessary to create a Virtual Service to route to this pod at the expected path:

cat << EOF | kubectl apply -f -
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: letsencrypt
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - $(glooctl proxy address | cut -f 1 -d ':').nip.io
    routes:
    - matchers:
      - exact: /.well-known/acme-challenge/q5x9q1C4pPg1RtDEiXK9aMAb9ExpepU4Pp14pGKDPXo
      routeAction:
        single:
          upstream:
            name: default-cm-acme-http-solver-f6mdb-8089
            namespace: gloo-system
EOF

Note that we are specifying the domain to be our nip.io domain and routing requests for the path that Let’s Encrypt expects, /.well-known/acme-challenge/<TOKEN> to the correct Upstream.

Now that the server can successfully reach the pod, the challenge should be complete, and our Certificate will be available for use:

% kubectl get certificates.cert-manager.io
NAME     READY   SECRET       AGE
nip-io   True    nip-io-tls   10m

Test

First, let’s make sure we have the petstore application installed on our cluster:

kubectl apply -f https://raw.githubusercontent.com/solo-io/gloo/v1.11.x/example/petstore/petstore.yaml

Then, we configure our Virtual Service to use our newly created TLS secret and route to the petstore application:

cat << EOF | kubectl apply -f -
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: letsencrypt
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - $(glooctl proxy address | cut -f 1 -d ':').nip.io
    routes:
    - matchers:
      - prefix: /
      routeAction:
        single:
          upstream:
            name: default-petstore-8080
            namespace: gloo-system
  sslConfig:
    secretRef:
      name: nip-io-tls
      namespace: default
EOF

Now we can curl the service:

% curl https://$(glooctl proxy address | cut -f 1 -d ':').nip.io/api/pets -k
[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]

Notice we use the -k flag so curl will not verify the certificate, which is necessary as the certificate we generated was from Let’s Encrypt’s “staging” CA, which is not trusted by our system.

Finally, we can inspect the certificate being presented by Envoy for this route:

% openssl s_client -connect $(glooctl proxy address | cut -f 1 -d ':').nip.io:443

You should see information regarding the certificate used for this connection. Specifically, you should see something similar to the following:

subject=/CN=34.71.xx.xx.nip.io
issuer=/CN=Fake LE Intermediate X1

We have just confirmed that the service is accessible over the HTTPS port and the certificate from Let’s Encrypt has been presented!

Integrating Gloo Edge and Let's Encrypt with cert-manager

Table of Contents

Prerequisites

Install Gloo Edge

Install cert manager

Utilizing the ACME DNS-01 Challenge

Using an AWS key pair

Provide AWS account details to cert-manager

Create a cluster issuer

Using AWS IRSA

Create role and policy

Create a cluster issuer

Check the cluster issuer state

Create a certificate for our service

Expose the service securly via Gloo Edge

Test!

Utilizing the ACME HTTP-01 Challenge

Create an Issuer

Create the Certificate

Routing to the cert-manager pod

Test