BYO Redis for extensions

Replace the managed ext-cache Redis with your own Redis instance for the rate limiter and external auth extensions.

About the built-in Redis instance

When you enable the rate limiter or external authentication extensions, Solo Enterprise for kgateway automatically deploys a single shared Redis®* ¹ instance. By default, the Redis instance is named ext-cache and is deployed per GatewayClass. It acts as the backing cache for both the rate limiter and the external auth service.

The following data is stored in the ext-cache Redis instance:

Replace the built-in Redis

You can replace the managed ext-cache instance with your own external Redis-compatible datastore. For example, you might want to implement one of the following use cases:

• Use a managed Redis service, such as AWS ElastiCache (Valkey or Redis OSS), GCP Memorystore, or Azure Cache for Redis
• Deploy a Redis cluster with a custom high-availability topology
• Share a single Redis instance across multiple gateways
• Meet compliance or data-residency requirements for session storage

The rate limiter and external auth service are configured independently and can point to different Redis instances or different databases on the same instance. You can also configure one component to use the bundled, managed ext-cache instance while the other one uses an external Redis service.

Before you begin

1. Create or use an existing EKS cluster. For more information, see the Amazon EKS documentation.
2. Follow the Get started guide to install Solo Enterprise for kgateway.
3. Follow the Sample app guide to create a gateway proxy with an HTTP listener and deploy the httpbin sample app.
4. Create a Redis instance that is external to your cluster environment, such as AWS ElastiCache. Make sure that the instance is in the same VPC as your cluster and that traffic is allowed between the cluster’s subnet and the subnet that the Redis instance is in.
5. Get the endpoint of your Redis instance in host:port format. For example, when you use an AWS ElastiCache instance with Valkey, your endpoint looks similar to the following: docs-aaa1aa.serverless.usw2.cache.amazonaws.com:6379.
6. Set up a user for your Redis instance. If you use Amazon ElastiCache with Valkey, a default database user is set up automatically.
7. If your Redis instance requires authentication or TLS, make sure that you have the required credentials and certificates to successfully connect to your instance.

Step 1: Set up AWS IAM roles and permissions

Set up IAM Roles for Service Accounts (IRSA) so that the rate limiter and external auth extension pods can authenticate to AWS ElastiCache by using short-lived IAM tokens instead of static credentials. You create an IAM role with a trust policy that allows the extension service accounts to assume it, attach a permission policy granting elasticache:Connect, and annotate the Kubernetes service accounts so the pods receive AWS credentials at runtime via the EKS OIDC provider.

1. Set environment variables for your environment. These variables are used in subsequent commands.

   # AWS account and cluster details
   export AWS_REGION=<aws-region>
   export AWS_CLUSTER_NAME=<eks-cluster-name>
   export AWS_ACCOUNT_ID=<aws-account-ID>
   
   # The AWS ElastiCache host name (host:port format), cache name and database user
   export AWS_ELASTICACHE_HOST=<aws-elasticache-host>
   export CACHE_NAME=<aws-cache-name>
   export DATABASE_USER=<db-user>
   
   # The service accounts to use for the extauth and rate limiter pods
   export EXTAUTH_SA=ext-auth-irsa-sa
   export RATELIMIT_SA=rate-limiter-irsa-sa

2. Get the OIDC provider URL for your EKS cluster. With IRSA, the extension pods exchange their Kubernetes service account tokens for temporary AWS credentials via this OIDC provider. The URL is later used in the trust policy to scope which pods can assume the role.

   OIDC_PROVIDER=$(aws eks describe-cluster \
     --name $AWS_CLUSTER_NAME \
     --region $AWS_REGION \
     --query "cluster.identity.oidc.issuer" \
     --output text | sed 's|https://||')
   
   echo $OIDC_PROVIDER

3. Create the trust policy. This policy defines the Kubernetes service accounts that are allowed to assume the IAM role. The following example scopes access to the service accounts that you create in a later step.

   cat > trust-policy.json <<EOF
   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Principal": {
         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity",
       "Condition": {
         "StringEquals": {
           "${OIDC_PROVIDER}:sub": [
             "system:serviceaccount:kgateway-system:${EXTAUTH_SA}",
             "system:serviceaccount:kgateway-system:${RATELIMIT_SA}"
           ]
         }
       }
     }]
   }
   EOF

4. Create the IAM role with the trust policy that you defined in the previous step.

   aws iam create-role \
     --role-name kgateway-elasticache \
     --assume-role-policy-document file://trust-policy.json

5. Create the permission policy file. This policy grants elasticache:Connect permissions to both the ElastiCache cache resource and the ElastiCache database user. ElastiCache validates the token against the user’s IAM auth configuration and the cache resource ID.

   cat > elasticache-policy.json <<EOF
   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": "elasticache:Connect",
       "Resource": [
         "arn:aws:elasticache:${AWS_REGION}:${AWS_ACCOUNT_ID}:serverlesscache:${CACHE_NAME}",
         "arn:aws:elasticache:${AWS_REGION}:${AWS_ACCOUNT_ID}:user:${DATABASE_USER}"
       ]
     }]
   }
   EOF
   info

   For provisioned ElastiCache clusters, replace serverlesscache with replicationgroup in the resource ARN.

6. Attach the permission policy to the IAM role.

   aws iam put-role-policy \
     --role-name kgateway-elasticache \
     --policy-name ElastiCacheConnect \
     --policy-document file://elasticache-policy.json

7. Get the role ARN and store it in an environment variable. You need this ARN to annotate the Kubernetes service accounts in the next step.

   export ROLE_ARN=$(aws iam get-role \
     --role-name kgateway-elasticache \
     --query "Role.Arn" \
     --output text)
   
   echo $ROLE_ARN

8. Create service accounts for the rate limiter and external auth extensions and annotate them with the IAM role ARN. You can also annote the service accounts that are created by default. For more information, see Use default service accounts for IRSA.

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: rate-limiter-irsa-sa
     namespace: kgateway-system
     annotations:
       eks.amazonaws.com/role-arn: "${ROLE_ARN}"
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: ext-auth-irsa-sa
     namespace: kgateway-system
     annotations:
       eks.amazonaws.com/role-arn: "${ROLE_ARN}"
   EOF

Step 2: Configure the Redis connection

Create an EnterpriseKgatewayParameters resource that disables the built-in ext-cache instance and configures the connection to your external Redis instance. Setting extCache.enabled: false prevents the managed ext-cache Deployment, Service, and ServiceAccount from being created. You can configure the rate limiter and external auth service independently. They can both share the same Redis instance or use different ones. You can also configure one component to use the built-in ext-cache Redis instance while the other one uses an external Redis instance.

1. Create an EnterpriseKgatewayParameters resource that configures your Redis connection.

   kubectl apply -f - <<EOF
   apiVersion: enterprisekgateway.solo.io/v1alpha1
   kind: EnterpriseKgatewayParameters
   metadata:
     name: redis
     namespace: kgateway-system
   spec:
     kube:
       sharedExtensions:
         extCache:
           enabled: false
         ratelimiter:
           enabled: true
           serviceAccountName: rate-limiter-irsa-sa
           redis:
             address: "${AWS_ELASTICACHE_HOST}"
             socketType: tls
             auth:
               aws:
                 region: "${AWS_REGION}"
                 clusterName: "${CACHE_NAME}"
                 userName: "${DATABASE_USER}"
                 serverlessCacheName: "${CACHE_NAME}"   # Same value as clusterName; required for Serverless only.
         extauth:
           enabled: true
           serviceAccountName: ext-auth-irsa-sa
           sessionRedis:
             address: "${AWS_ELASTICACHE_HOST}"
             socketType: tls
             auth:
               aws:
                 region: "${AWS_REGION}"
                 clusterName: "${CACHE_NAME}"
                 userName: "${DATABASE_USER}"
                 serverlessCacheName: "${CACHE_NAME}"   # Same value as clusterName; required for Serverless only.
   EOF

   Setting	Description
   extCache.enabled	Disable the default ext-cache Redis instance so that you can configure your own.
   address	The address of your Redis instance in host:port format.
   socketType	Set to TLS to connect to your Redis instance by using a TLS connection with public certs. If your Redis instance is set up with a private certificate, you must store the certificate in a Kubernetes secret. For more information, see Use a private CA certificate for TLS.
   auth.aws.region	Scope the token signature to the correct AWS region
   auth.aws.clusterName	The resource identifier that is used in the IAM token signature. For provisioned clusters, this is the replication group ID. For Serverless caches (including Valkey), this is the cache name. . To find this value, use the following paths in the AWS console: Provisioned: AWS Console → ElastiCache → Clusters → Cluster ID. Serverless: AWS Console → ElastiCache → Valkey caches (or Redis OSS caches) → Cache name.
   auth.aws.userName	The ID of an ElastiCache database user that has IAM as its authentication type enabled. The IAM token is minted for this specific user. To find this value, use the following path in AWS console: AWS Console → ElastiCache → User management → your user → User ID
   auth.aws.serverlessCacheName	ElastiCache Serverless (including Valkey): Also set serverlessCacheName to the same value as clusterName. The IAM token signature for serverless uses a different resource path than provisioned clusters, so both fields are required. If you use Redis with provisioned clusters, this field can be omitted.

2. Update your GatewayClass to reference the EnterpriseKgatewayParameters resource.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1
   kind: GatewayClass
   metadata:
     name: enterprise-kgateway
   spec:
     controllerName: solo.io/enterprise-kgateway
     parametersRef:
       group: enterprisekgateway.solo.io
       kind: EnterpriseKgatewayParameters
       name: redis
       namespace: kgateway-system
   EOF

3. Verify that the ext-auth-service and rate-limiter pods restart.

   kubectl get pods -n kgateway-system -l app=rate-limiter
   kubectl get pods -n kgateway-system -l app=ext-auth-service

Step 3: Verify the connection

To verify the connection, set up a rate limiting or an external auth policy for the httpbin sample app. Follow one of the following guides:

• Enterprise global rate limiting
• External auth with OIDC authorization code

Limitations

The following are not supported in this release:

• AWS ElastiCache memcached: Memcached is currently not supported. Use AWS ElastiCache with Valkey or Redis OSS instead.
• Redis Sentinel mode: The rate limiter and external auth service binaries do not support Sentinel. Use Redis Cluster mode (clustered: true) or a cloud-managed HA offering with automatic failover instead.
• AuthConfig overrides: When extauth.sessionRedis is set in the EnterpriseKgatewayParameters resource, AuthConfigs that omit the redis block entirely automatically inherit all server-level Redis settings. AuthConfigs that set a redis block inherit the host and tlsCertMountPath of the EnterpriseKgatewayParameters resource. However, you can override the db, poolSize, and socketType fields in the AuthConfig. Redis credentials can only be set at the server level; the AuthConfig RedisOptions proto has no auth fields, so credentials cannot be overridden per-AuthConfig.
• Credential sources: Only Kubernetes Secrets, AWS IRSA, and AWS EKS Pod Identity are supported as credential sources. Other sources, such as HashiCorp Vault and AWS Secrets Manager, are not supported.

Other configurations

Review other common Redis configurations.

Use default service accounts for IRSA

Instead of creating service accounts for the extauth and rate limiter pods, you can annotate the auto-generated service accounts instead. Note that the recommended approach is to use your own service accounts so that you have control over the service account lifecycle.

1. Annotate the rate limiter and external auth service accounts with the IAM role ARN. This way, the pod receives AWS credentials and can successfully authenticate and generate ElastiCache IAM tokens.

   kubectl annotate sa -n kgateway-system \
     rate-limiter-kgateway-system-${GATEWAYCLASS_NAME} \
     "eks.amazonaws.com/role-arn=${ROLE_ARN}" --overwrite
   
   kubectl annotate sa -n kgateway-system \
     ext-auth-service-${GATEWAYCLASS_NAME} \
     "eks.amazonaws.com/role-arn=${ROLE_ARN}" --overwrite

2. Restart the extension pods to pick up the new service account annotation.

   kubectl -n kgateway-system rollout restart deploy -l app=rate-limiter
   kubectl -n kgateway-system rollout restart deploy -l app=ext-auth-service
   kubectl -n kgateway-system rollout status deploy

3. Continue with configuring your EnterpriseKgatewayParameters resource. Omit the serviceAccountName field as shown in this example configuration.

   kubectl apply -f - <<EOF
   apiVersion: enterprisekgateway.solo.io/v1alpha1
   kind: EnterpriseKgatewayParameters
   metadata:
     name: redis
     namespace: kgateway-system
   spec:
     kube:
       sharedExtensions:
         extCache:
           enabled: false
         ratelimiter:
           enabled: true
           redis:
             address: "${AWS_ELASTICACHE_HOST}"
             socketType: tls
             auth:
               aws:
                 region: "${AWS_REGION}"
                 clusterName: "${CACHE_NAME}"
                 userName: "${DATABASE_USER}"
                 serverlessCacheName: "${CACHE_NAME}"   # Required for Serverless only.
         extauth:
           enabled: true
           serviceAccountName: ext-auth-irsa-sa
           sessionRedis:
             address: "${AWS_ELASTICACHE_HOST}"
             socketType: tls
             auth:
               aws:
                 region: "${AWS_REGION}"
                 clusterName: "${CACHE_NAME}"
                 userName: "${DATABASE_USER}"
                 serverlessCacheName: "${CACHE_NAME}"   # Required for Serverless only.
   EOF

Use username/password authentication

If your Redis instance requires username and password authentication instead of IAM, store the credentials in a Kubernetes Secret and reference them in your EnterpriseKgatewayParameters resource.

1. Create a Kubernetes Secret with your Redis credentials in the gateway namespace.

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: redis-credentials
     namespace: kgateway-system
   stringData:
     redis-username: "<your-redis-username>"   # Leave empty if no username is required.
     redis-password: "<your-redis-password>"
   EOF

2. Reference the Secret in your EnterpriseKgatewayParameters resource by using the auth.secretRef field.

   kubectl apply -f - <<EOF
   apiVersion: enterprisekgateway.solo.io/v1alpha1
   kind: EnterpriseKgatewayParameters
   metadata:
     name: redis
     namespace: kgateway-system
   spec:
     kube:
       sharedExtensions:
         extCache:
           enabled: false
         ratelimiter:
           enabled: true
           redis:
             address: "${AWS_ELASTICACHE_HOST}"
             auth:
               secretRef:
                 name: redis-credentials
                 passwordKey: "redis-password"
                 usernameKey: "redis-username"
         extauth:
           enabled: true
           sessionRedis:
             address: "${AWS_ELASTICACHE_HOST}"
             auth:
               secretRef:
                 name: redis-credentials
                 passwordKey: "redis-password"
                 usernameKey: "redis-username"
   EOF

Use a private CA certificate for TLS

If your Redis instance uses a certificate that is signed by a private or self-signed CA, store the CA certificate in a Kubernetes Secret and reference it in your EnterpriseKgatewayParameters resource.

1. Store the CA certificate in a Kubernetes Secret in the gateway namespace.

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: redis-tls
     namespace: kgateway-system
   data:
     ca.crt: $(cat ca.crt | base64 | tr -d '\n')
   EOF

2. Reference the Secret in your EnterpriseKgatewayParameters resource by using the certs.caCertSecretRef field.

   kubectl apply -f - <<EOF
   apiVersion: enterprisekgateway.solo.io/v1alpha1
   kind: EnterpriseKgatewayParameters
   metadata:
     name: redis
     namespace: kgateway-system
   spec:
     kube:
       sharedExtensions:
         extCache:
           enabled: false
         ratelimiter:
           enabled: true
           redis:
             address: "${AWS_ELASTICACHE_HOST}"
             socketType: tls
             certs:
               caCertSecretRef:
                 name: redis-tls
                 namespace: kgateway-system
               caCertKey: ca.crt
         extauth:
           enabled: true
           sessionRedis:
             address: "${AWS_ELASTICACHE_HOST}"
             socketType: tls
             certs:
               caCertSecretRef:
                 name: redis-tls
                 namespace: kgateway-system
               caCertKey: ca.crt
   EOF

Use ext-cache for one component and custom Redis for another

You can keep the built-in ext-cache instance for one component while routing the other component to an external Redis instance. This is useful when, for example, you want to use your own Redis for external auth session storage but are happy to let the rate limiter continue using the auto-provisioned ext-cache.

In this mode, extCache.enabled remains true (the default). You only configure an external Redis address for the component that should use it. The other component automatically continues to use ext-cache.

The following example uses the default ext-cache Redis instance for the rate limiter, but configures a custom instance for the extauth service.

kubectl apply -f - <<EOF
apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayParameters
metadata:
  name: redis
  namespace: kgateway-system
spec:
  kube:
    sharedExtensions:
      extCache:
        enabled: true
      ratelimiter:
        enabled: true
      extauth:
        enabled: true
        sessionRedis:
          address: "${AWS_ELASTICACHE_HOST}"
          socketType: tls
          certs:
            caCertSecretRef:
              name: redis-tls
              namespace: kgateway-system
            caCertKey: ca.crt
EOF

Redis connection pool tuning

You can tune the Redis connection pool and timeouts by using the connection field. When not set, the Redis client library defaults apply.

External auth

Add the connection field under the sessionRedis field in your EnterpriseKgatewayParameters resource.

sharedExtensions:
  extauth:
    sessionRedis:
      address: "${AWS_ELASTICACHE_HOST}"
      connection:
        poolSize: 10
        minIdleConns: 2
        maxIdleConns: 5
        dialTimeout: 5s

Rate limiter

Add the connection field under the redis field in your EnterpriseKgatewayParameters resource.

sharedExtensions:
  ratelimiter:
    redis:
      address: "${AWS_ELASTICACHE_HOST}"
      connection:
        poolSize: 10
        minIdleConns: 2
        maxIdleConns: 5
        dialTimeout: 5s