Configure cross-account Lambda access
Configure Gloo Gateway to route to AWS Lambda functions in different accounts than the AWS account that Gloo Gateway authenticates with.
About
In some cases, you might want Gloo Gateway to route requests to Lambda functions that exist in other AWS accounts that the account the you used to authenticate with Gloo Gateway. For example, your organization might establish multitenancy in which each team has its own AWS account where its Lambda functions exist, but your organization uses one dedicated AWS account for authentication purposes.
You can set up multi-account routing in the following ways:
- IAM Roles for Service Accounts role-chained configuration (recommended): Use AWS IAM Roles for Service Accounts (IRSA) to configure routing to functions in different accounts. This method is recommended for routing to Lambda functions across multiple accounts.
- Resource-based configuration: Use AWS resource-based configuration to enable routing to functions in different accounts. A resource-based policy can give other accounts the permission to invoke a function without requiring the other accounts to assume a role.
IRSA role-chained configuration
Use AWS IAM Roles for Service Accounts (IRSA) to configure routing to functions in different accounts. This method is recommended for using cross-account Lambda functions with Gloo Gateway.
Review the following diagram to understand the role-chaining flow of this method.
flowchart LR
A[Gateway proxy<br>Service Account] -->|assume via STS| B[/"Gateway<br>Cross-account auth role<br>(gloo-lambda-auth-role)"/]
B -->|assume via STS| C[/"Upstream<br>Invocation role<br>(gloo-lambda-invoke-role)"/]
C -->|invoke| D[AWS Lambda function]
This method involves defining two roles that the gateway proxy service account assumes via IRSA:
- Authenticate: In the account that you want to use to authenticate with AWS (the “authentication account”), you create a role that the gateway proxy service account assumes to securely access the Lambda account. You then annotate the gateway proxy service account so that it can assume this role via IRSA.
- Invoke: In the account that contains the Lambda functions you want to route to (the “Lambda account”), you create a role that the gateway’s assumed authentication role can then use to invoke individual Lambda functions. You define this role in an Upstream resource that you configure for the Lambda functions that you want to invoke.
This method requires you to enable IAM settings in your EKS cluster, such as the AWS Pod Identity Webhook, before you deploy Gloo Gateway components that are created during the installation, such as the Gateway CRD and the gateway proxy service account. You might use these steps with a fresh EKS test cluster to try out IRSA role-chained configuration with Gloo Gateway.
Save AWS account details
Save the ID of the AWS account that you want to use for authentication (the “authentication account”).
export AUTH_ACCOUNT_ID=<auth_account_id>Save the region and ID of the AWS account where the Lambda functions that you want to invoke exist (the “Lambda account”).
export AWS_LAMBDA_REGION=<lambda_account_region> export LAMBDA_ACCOUNT_ID=<lambda_account_id>Save the region and name of the EKS cluster that you plan to install Gloo Gateway in.
export AWS_CLUSTER_REGION=<cluster_region> export CLUSTER_NAME=<cluster_name>Check whether your EKS cluster has an OIDC provider.
export OIDC_PROVIDER=$(aws eks describe-cluster --name ${CLUSTER_NAME} --region ${AWS_CLUSTER_REGION} --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") echo $OIDC_PROVIDER- If an OIDC provider in the format
oidc.eks.<region>.amazonaws.com/id/<cluster_id>is returned, continue to the next section. - If an OIDC provider is not returned, follow the AWS documentation to Create an IAM OIDC provider for your cluster, and then run this command again to save the OIDC provider in an environment variable.
- If an OIDC provider in the format
Configure AWS IAM resources
Create roles in your authentication and Lambda AWS accounts.
- In the account that you want to use to authenticate with AWS (the “authentication account”), you create a
gloo-lambda-auth-rolethat the gateway proxy service account assumes to securely access the Lambda account. In later steps, you annotate thegloo-proxy-httpservice account so that it can assume this role via IRSA. - In the account that contains the Lambda functions you want to route to (the “Lambda account”), you create a
gloo-lambda-invoke-rolethat the gateway’s assumed authentication role can then use to invoke the Lambda functions. In later steps, you define this role in an Upstream resource that you configure for the Lambda functions you want to invoke.
In your authentication account, create the following resources.
Create an IAM policy to allow access to the following four Lambda actions.
cat >policy-1.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:ListFunctions", "lambda:InvokeFunction", "lambda:GetFunction", "lambda:InvokeAsync" ], "Resource": "*" } ] } EOF aws iam create-policy --policy-name gloo-lambda-auth-policy --policy-document file://policy-1.jsonCreate the following IAM role, which associates the policy with the Kubernetes service account of the HTTP gateway proxy component. Note that the service account name
gloo-proxy-httpin thegloo-systemnamespace is specified, because in later steps you create an HTTP gateway namedhttp.cat >role-1.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AUTH_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_PROVIDER}:sub": [ "system:serviceaccount:gloo-system:gloo-proxy-http" ] } } } ] } EOF aws iam create-role --role-name gloo-lambda-auth-role --assume-role-policy-document file://role-1.jsonAttach the IAM role to the IAM policy. This IAM role for the service account is known as an IRSA.
aws iam attach-role-policy --role-name gloo-lambda-auth-role --policy-arn=arn:aws:iam::${AUTH_ACCOUNT_ID}:policy/gloo-lambda-auth-policyVerify that the policy is attached to the role.
aws iam list-attached-role-policies --role-name gloo-lambda-auth-roleExample output:
{ "AttachedPolicies": [ { "PolicyName": "gloo-lambda-auth-policy", "PolicyArn": "arn:aws:iam::111122223333:policy/gloo-lambda-auth-policy" } ] }
In your Lambda account, create the following resources.
Create an IAM policy that contains at least the
lambda:InvokeFunctionpermission to allow Lambda function invocation, such as the following example.cat >policy-2.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "lambda:ListFunctions", "lambda:InvokeFunction", "lambda:GetFunction", "lambda:InvokeAsync" ], "Resource": "*" } ] } EOF aws iam create-policy --policy-name gloo-lambda-invoke-policy --policy-document file://policy-2.jsonCreate an IAM role that specifies the ARN of the authentication account’s role that you created in step 1.2, such as the following example. This ensures that the authentication account’s role that you previously created can assume this Lambda account role to invoke functions.
cat >role-2.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::${AUTH_ACCOUNT_ID}:role/gloo-lambda-auth-role" }, "Action": "sts:AssumeRole", "Condition": {} } ] } EOF aws iam create-role --role-name gloo-lambda-invoke-role --assume-role-policy-document file://role-2.jsonAttach the invocation IAM role to the invocation IAM policy.
aws iam attach-role-policy --role-name gloo-lambda-invoke-role --policy-arn=arn:aws:iam::${LAMBDA_ACCOUNT_ID}:policy/gloo-lambda-invoke-policyVerify that the policy is attached to the role.
aws iam list-attached-role-policies --role-name gloo-lambda-invoke-roleExample output:
{ "AttachedPolicies": [ { "PolicyName": "gloo-lambda-invoke-policy", "PolicyArn": "arn:aws:iam::111122223333:policy/gloo-lambda-invoke-policy" } ] }Choose an existing or create a new Lambda function that you want to route to. To create a simple
echoNode.js function for testing, see Create a Lambda function for testing.
Deploy the Amazon EKS Pod Identity Webhook
Before you install Gloo Gateway, deploy the Amazon EKS Pod Identity Webhook, which allows pods’ service accounts to use AWS IAM roles. When you create the Gloo Gateway proxy in the next section, this webhook mutates the proxy’s service account so that it can assume your IAM role to invoke Lambda functions.
In your EKS cluster, install cert-manager, which is a prerequisite for the webhook.
wget https://github.com/cert-manager/cert-manager/releases/download/v1.12.4/cert-manager.yaml kubectl apply -f cert-manager.yamlVerify that all cert-manager pods are running.
kubectl get pods -n cert-managerDeploy the Amazon EKS Pod Identity Webhook.
kubectl apply -f https://raw.githubusercontent.com/solo-io/workshops/refs/heads/master/gloo-gateway/1-18/enterprise/lambda/data/steps/deploy-amazon-pod-identity-webhook/auth.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/workshops/refs/heads/master/gloo-gateway/1-18/enterprise/lambda/data/steps/deploy-amazon-pod-identity-webhook/deployment-base.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/workshops/refs/heads/master/gloo-gateway/1-18/enterprise/lambda/data/steps/deploy-amazon-pod-identity-webhook/mutatingwebhook.yaml kubectl apply -f https://raw.githubusercontent.com/solo-io/workshops/refs/heads/master/gloo-gateway/1-18/enterprise/lambda/data/steps/deploy-amazon-pod-identity-webhook/service.yamlVerify that the webhook deployment completes.
kubectl rollout status deploy/pod-identity-webhook
Install Gloo Gateway with service account authentication
Install the open source or enterprise edition of Gloo Gateway, including settings to support gateway proxy service account authentication with your AWS account.
Open source
Install the custom resources of the Kubernetes Gateway API.
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yamlExample output:
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io createdAdd the Helm repository for Gloo Gateway Open Source.
helm repo add gloo https://storage.googleapis.com/solo-public-helm helm repo updateInstall Gloo Gateway. This command creates the
gloo-systemnamespace and installs the Gloo Gateway control plane into it. The following settings also enable Gloo Gateway to use an AWS IRSA and annotate thegloo-proxy-httpservice account with thegloo-lambda-invoke-role.helm install -n gloo-system gloo gloo/gloo \ --create-namespace \ --version 1.19.8 \ -f -<<EOF discovery: enabled: false gatewayProxies: gatewayProxy: disabled: true gloo: disableLeaderElection: true kubeGateway: enabled: true # Annotate the gateway proxy service account gatewayParameters: glooGateway: serviceAccount: extraAnnotations: eks.amazonaws.com/role-arn: arn:aws:iam::${LAMBDA_ACCOUNT_ID}:role/gloo-lambda-invoke-role # Enable AWS account authentication with IRSA settings: aws: enableServiceAccountCredentials: true stsCredentialsRegion: ${AWS_LAMBDA_REGION} EOFExample output:
NAME: gloo-gateway LAST DEPLOYED: Thu Apr 18 11:50:39 2024 NAMESPACE: gloo-system STATUS: deployed REVISION: 2 TEST SUITE: NoneVerify that the Gloo Gateway control plane is up and running.
kubectl get pods -n gloo-system | grep glooExample output:
NAME READY STATUS RESTARTS AGE gloo-78658959cd-cz6jt 1/1 Running 0 12s
Enterprise edition
Set your Gloo Gateway license key as an environment variable. If you do not have one, contact an account representative.
export GLOO_GATEWAY_LICENSE_KEY=<license-key>Install the custom resources of the Kubernetes Gateway API.
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yamlExample output:
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created customresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io createdAdd the Helm repository for Gloo Gateway Enterprise Edition.
helm repo add glooe https://storage.googleapis.com/gloo-ee-helm helm repo updateInstall Gloo Gateway. This command creates the
gloo-systemnamespace and installs the Gloo Gateway control plane into it. The following settings also enable Gloo Gateway to use an AWS IRSA and annotate the gateway proxy service account with thegloo-lambda-invoke-role.helm install -n gloo-system gloo glooe/gloo-ee \ --create-namespace \ --version 1.19.11 \ --set-string license_key=$GLOO_GATEWAY_LICENSE_KEY \ -f -<<EOF gloo: discovery: enabled: false gatewayProxies: gatewayProxy: disabled: true kubeGateway: enabled: true # Annotate the gateway proxy service account gatewayParameters: glooGateway: serviceAccount: extraAnnotations: eks.amazonaws.com/role-arn: arn:aws:iam::${LAMBDA_ACCOUNT_ID}:role/gloo-lambda-invoke-role gloo: disableLeaderElection: true # Enable AWS account authentication with IRSA settings: aws: enableServiceAccountCredentials: true stsCredentialsRegion: ${AWS_LAMBDA_REGION} gloo-fed: enabled: false glooFedApiserver: enable: false grafana: defaultInstallationEnabled: false observability: enabled: false prometheus: enabled: false EOFExample output:
NAME: gloo-gateway LAST DEPLOYED: Thu Apr 18 11:50:39 2024 NAMESPACE: gloo-system STATUS: deployed REVISION: 2 TEST SUITE: NoneVerify that the Gloo Gateway control plane is up and running.
kubectl get pods -n gloo-systemExample output:
NAME READY STATUS RESTARTS AGE extauth-64458459c8-9q9kq 1/1 Running 0 26s gateway-certgen-qpv5q 0/1 Completed 0 31s gloo-68846f8459-nbzxp 1/1 Running 0 26s gloo-resource-rollout-check-xskxt 0/1 Completed 0 25s gloo-resource-rollout-n9kvf 0/1 Completed 0 26s rate-limit-5d8f859465-qhgsg 1/1 Running 0 26s redis-9f4f98445-5fh69 1/1 Running 0 26s
Create a gateway proxy and annotate its service account
Create an HTTP gateway, and verify that its service account is annotated.
Create a Gateway resource and configure an HTTP listener.
kubectl apply -n gloo-system -f- <<EOF kind: Gateway apiVersion: gateway.networking.k8s.io/v1 metadata: name: http spec: gatewayClassName: gloo-gateway listeners: - protocol: HTTP port: 8080 name: http allowedRoutes: namespaces: from: All EOFVerify that the gateway is created successfully. You can also review the external address that is assigned to the gateway. Note that depending on your environment it might take a few minutes for the load balancer service to be assigned an external address.
kubectl get gateway http -n gloo-systemExample output:
NAME CLASS ADDRESS PROGRAMMED AGE http gloo-gateway a3a6c06e2f4154185bf3f8af46abf22e-139567718.us-east-2.elb.amazonaws.com True 93sGet the external address of the gateway and save it in an environment variable.
Verify that the
gloo-proxy-httpservice account has theeks.amazonaws.com/role-arn: arn:aws:iam::${LAMBDA_ACCOUNT_ID}:role/gloo-lambda-invoke-roleannotation.kubectl describe serviceaccount gloo-proxy-http -n gloo-system
Set up routing to your function
Create Upstream and HTTPRoute resources to route requests to the Lambda function.
Create an Upstream resource for the Lambda functions that you want to route to. You can specify multiple functions as a list in the
spec.aws.lambdaFunctionsarray.kubectl apply -f - <<EOF apiVersion: gloo.solo.io/v1 kind: Upstream metadata: name: lambda-upstream namespace: gloo-system spec: aws: region: ${AWS_LAMBDA_REGION} roleArn: arn:aws:iam::${LAMBDA_ACCOUNT_ID}:role/gloo-lambda-invoke-role lambdaFunctions: - lambdaFunctionName: <function_name> qualifier: $LATEST # logicalName: <function_name:qualifier> EOFField Type Description lambdaFunctionNamestringThe name of the Lambda function as it appears in the AWS Lambda portal. qualifierstringThe version qualifier for the Lambda function, such as $LATEST. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.htmllogicalNamestringOptional: A logical name that Gloo Gateway associates with this function. If unset, defaults to <lambdaFunctionName>:<qualifier>.
Resource-based configuration
Use AWS resource-based configuration to configure routing to functions in different accounts. The resource-based policy can give other accounts the permission to invoke the function, without requiring the other account to assume a role.
Before you begin
Follow the Get started guide to install Gloo Gateway, set up a gateway resource, and deploy the httpbin sample app.
Get the external address of the gateway and save it in an environment variable.
AWS resources
For the AWS configuration, you create a user or role in the authentication account, and a Lambda function in the account that contains the Lambda functions. The Lambda function has a resource-based policy statement which allows the user or role in the authentication account to invoke it.
In your authentication account, create the following resources.
- Create a user or role in your authentication account. Be sure to give the user or role the
lambda:InvokeFunctionpermission, so that the role can be used to invoke the Lambda functions in the other account. - Create an access key for the user or role, which is used to authenticate with AWS when invoking the Lambda functions.
- Create a Kubernetes secret that contains the access key and secret key.
glooctl create secret aws \ --name 'aws-creds' \ --namespace gloo-system \ --access-key ${AWS_ACCESS_KEY_ID} \ --secret-key ${AWS_SECRET_ACCESS_KEY}
- Create a user or role in your authentication account. Be sure to give the user or role the
In your Lambda account, create the following resources.
- Choose an existing or create a new Lambda function that you want to route to. To create a simple
echoNode.js function for testing, see Create a Lambda function for testing. - Define a resource-based policy statement for the function, which allows the user in the authentication account to invoke it.
- In the AWS console, select the Lambda function.
- Click the Configuration tab.
- In the sidebar, click the Permissions tab.
- In the Resource-based policy statements section, click Add Permissions.
- Select AWS account as the entity which invokes the function.
- Specify the ARN of the user or role in the authentication account as the principal.
- Select
lambda:InvokeFunctionas the action.
- Choose an existing or create a new Lambda function that you want to route to. To create a simple
Gloo Gateway resources
Create Upstream resources to support routing to your Lambda functions.
Create an Upstream resource for the Lambda functions that you want to route to. You can specify multiple functions as a list in the
spec.aws.lambdaFunctionsarray.apiVersion: gloo.solo.io/v1 kind: Upstream metadata: name: lambda-upstream namespace: gloo-system spec: aws: region: ${AWS_LAMBDA_REGION} secretRef: name: aws-creds namespace: gloo-system awsAccountId: ${LAMBDA_ACCOUNT_ID} lambdaFunctions: - lambdaFunctionName: <function_name> qualifier: $LATEST # logicalName: <function_name:qualifier>Field Type Description lambdaFunctionNamestringThe name of the Lambda function as it appears in the AWS Lambda portal. qualifierstringThe version qualifier for the Lambda function, such as $LATEST. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.htmllogicalNamestringOptional: A logical name that Gloo Gateway associates with this function. If unset, defaults to <lambdaFunctionName>:<qualifier>.
Verify routing
Create an HTTPRoute resource that references the
lambda-upstreamUpstream.- In the
rules.matchessection, specify a request matcher for your Lambda function, such as aPathPrefixmatch to a specific path. - In the
backendRefs.filters.extensionRef.namefield, specify your Lambda function name. ThisextensionReftells Gloo Gateway which function in the Upstream to reference, because multiple functions can be listed in thespec.aws.lambdaFunctionsarray of the Upstream. To route to another function that you listed in the same Upstream, you can either add anotherruleentry in this same HTTPRoute that references the same Upstream with a differentextensionReffor the function name, or create a new HTTPRoute to define the routing rule.
You cannot currently attach aprefixRewritepolicy to a Lambda Upstream.kubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: lambda-httproute namespace: gloo-system spec: parentRefs: - name: http namespace: gloo-system rules: - matches: - path: type: PathPrefix value: /<path> backendRefs: - name: lambda-upstream namespace: gloo-system group: gloo.solo.io kind: Upstream filters: - type: ExtensionRef extensionRef: group: "gloo.solo.io" kind: Parameter name: <function_name> EOF- In the
Confirm that Gloo Gateway correctly routes requests to Lambda by sending a curl request, such as this example command. Note that if you used IRSA role-chained configuration, the first request might take a few seconds to process, because the STS credential request must be performed first. However, after the credentials are cached, subsequent requests are processed more quickly.
Cleanup
You can optionally remove the resources that you set up as part of this guide.
Routing resources: Delete the lambda-httproute and lambda-upstream resources.
kubectl delete HTTPRoute lambda-httproute -n gloo-system
kubectl delete Upstream lambda-upstream -n gloo-system
Resource-based configuration only: Delete the aws-creds secret.
kubectl delete secret aws-creds -n gloo-system
IRSA role-chain configuration only: If you no longer need to access Lambda functions from Gloo Gateway:
Upgrade your Gloo Gateway Helm installation to remove the added settings and annotation.
- Get the Helm values for your current installation, and save them in a file.
helm get values gloo -n gloo-system -o yaml > gloo-gateway.yaml open gloo-gateway.yaml - Delete the following settings, and save the file. For Enterprise edition, these settings are in the
gloosection.kubeGateway.gatewayParameters.glooGateway.serviceAccount.extraAnnotationssettings.aws.enableServiceAccountCredentialssettings.aws.stsCredentialsRegion
- Upgrade the Helm release. Replace the example Gloo Gateway version with the version that you run.
- Open source:
helm upgrade -n gloo-system gloo gloo/gloo \ -f gloo-gateway.yaml \ --version=1.19.8 - Enterprise edition:
helm upgrade -n gloo-system gloo glooe/gloo-ee \ -f gloo-gateway.yaml \ --version=1.19.11
- Open source:
- Get the Helm values for your current installation, and save them in a file.
Delete the pod identity webhook.
kubectl delete deploy pod-identity-webhookRemove cert-manager.
kubectl delete -f cert-manager.yaml -n cert-manager kubectl delete ns cert-managerDelete the AWS IAM resources from the Lambda account.
aws iam detach-role-policy --role-name gloo-lambda-invoke-role --policy-arn=arn:aws:iam::${LAMBDA_ACCOUNT_ID}:policy/gloo-lambda-invoke-policy aws iam delete-role --role-name gloo-lambda-invoke-role aws iam delete-policy --policy-arn=arn:aws:iam::${LAMBDA_ACCOUNT_ID}:policy/gloo-lambda-invoke-policyDelete the AWS IAM resources from the authentication account.
aws iam detach-role-policy --role-name gloo-lambda-auth-role --policy-arn=arn:aws:iam::${AUTH_ACCOUNT_ID}:policy/gloo-lambda-auth-policy aws iam delete-role --role-name gloo-lambda-auth-role aws iam delete-policy --policy-arn=arn:aws:iam::${AUTH_ACCOUNT_ID}:policy/gloo-lambda-auth-policy