Skip to content
You are viewing the latest documentation for Solo Enterprise for kgateway. To access the documentation for older versions, use the version switcher.

OPA server as a sidecar

Page as Markdown
Enforce OPA policies through an OPA server that runs as a sidecar to the external auth service.
Solo Enterprise for kgateway can run your own Open Policy Agent (OPA) server as a sidecar to its built-in external auth service. You are responsible for administering the server per OPA best practices. You might choose this approach for larger scale environments and extended use cases, such as bundling and caching.
Because the OPA container is part of a Solo Enterprise for kgateway-managed pod, the OPA sidecar is subject to the lifecycle of the ext-auth deployment. If you want the OPA server to persist separately from Solo Enterprise for kgateway configuration, consider the Bring your own OPA server approach instead. In both cases, you are responsible for specifying the OPA configuration and Rego policies.
When running your OPA server as a sidecar, you can also leverage Rego rule bundling. With bundling, your Rego rules can live as a signed bundle in an external, central location, such as an AWS S3 bucket to meet your internal security requirements. This sidecar approach increases the resources that are needed in the external auth server pod, but works better at scale and provides more OPA-native support for teams that are familiar with administering an OPA server.
Architecture

The following diagram shows how the OPA server sidecar works with the external auth service.
    sequenceDiagram
    participant OPA as OPA sidecar
    
    Note over OPA: On startup: load Rego rules<br/>from mounted ConfigMap<br/>(or external bundle server)

    participant Client
    participant GW as Gateway
    participant EA as ext-auth-service
    participant UP as Upstream

    Client->>GW: 1. Request matches OPA-protected route
    GW->>EA: 2. Authorization request
    EA->>OPA: 3. Evaluate policy (localhost:8181)
    OPA->>OPA: 4. Check request against loaded rules
    OPA-->>EA: 5. Authorization decision
    EA-->>GW: 6. Forward decision

    alt Authorized (allow = true)
        GW->>UP: 7. Forward request to upstream
        UP-->>GW: Response
        GW-->>Client: 200 OK
    else Not Authorized (allow = false)
        GW-->>Client: 403 Forbidden
    end
  
Before you begin

  1. Follow the Get started guide to install Solo Enterprise for kgateway.
  2. Follow the Sample app guide to deploy the httpbin sample app. You do not need to create the Gateway or HTTPRoute from the sample app guide because the sidecar setup section creates its own.
Create the Rego rules

Create a Rego policy and store it in a Kubernetes config map. Because the deployment overlay mounts the config map directly into the ext-auth pod, the config map must exist in the same namespace as the ext-auth deployment. The config map name must match the configMap.name value in the deployment overlay volume that you create later.
  1. Create a Rego rule. The following policy checks for an allow: true request header. The policy returns a structured result object that includes the authorization decision and the HTTP status code for the response.
    cat <<EOF > policy.rego
package test
import future.keywords.if

default allow = false
allow if { input.http_request.headers["allow"] == "true" }

http_status := 200 if { allow }
http_status := 403 if { not allow }

result["allow"] := allow
result["http_status"] := http_status
EOF
    Review the following table to understand this configuration.
    SettingDescription
    default allow = falseDenies all requests by default.
    allow if {...}Allows requests that include the allow: true header.
    http_statusReturns a 200 status for allowed requests and 403 for denied requests.
    resultA structured object that packages the allow decision and http_status together. The AuthConfig queries the result rule to get this object.
    If you modify this policy, keep the package test declaration. The AuthConfig package field must match the package name in your Rego policy.
  2. Store the OPA policy in a Kubernetes config map in the same namespace as the ext-auth deployment.
    kubectl -n kgateway-system create configmap opa-policy --from-file=policy.rego
Enable the OPA server sidecar

Deploy the OPA server sidecar by adding it to the external auth service deployment through a EnterpriseKgatewayParameters overlay.
The following steps use a deployment overlay to add the OPA server as a sidecar container to the external auth service. For more information about customizing shared extension deployments, see Change shared extensions.
  1. Create a EnterpriseKgatewayParameters resource with a deployment overlay that adds the OPA sidecar container and the required volumes to the ext-auth deployment.
    kubectl apply -f - <<EOF
apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayParameters
metadata:
  name: opa-sidecar-params
  namespace: kgateway-system
spec:
  kube:
    sharedExtensions:
      extauth:
        enabled: true
        deploymentOverlay:
          spec:
            template:
              spec:
                volumes:
                - name: opa-policy
                  configMap:
                    name: opa-policy
                containers:
                - name: opa-auth
                  image: openpolicyagent/opa:0.70.0
                  args:
                  - "run"
                  - "--server"
                  - "--addr=0.0.0.0:8181"
                  - "--log-level=info"
                  - "--disable-telemetry"
                  - "/policies/policy.rego"
                  ports:
                  - containerPort: 8181
                  volumeMounts:
                  - name: opa-policy
                    mountPath: /policies
EOF
    Review the following table to understand this configuration.
    SettingDescription
    sharedExtensions.extauth.deploymentOverlayA strategic merge patch applied to the generated ext-auth Deployment. This overlay adds the OPA container and volumes.
    volumesThe opa-policy volume references the opa-policy ConfigMap that was created.
    volumeMountsMounts the ConfigMap volume into the container at /policies. Kubernetes creates a file for each key in the ConfigMap, so the policy.rego key becomes /policies/policy.rego inside the container.
    argsThe last argument (/policies/policy.rego) tells OPA to load the Rego policy from the mounted ConfigMap file. The --disable-telemetry flag prevents OPA from sending usage data to the OPA project, which is recommended for enterprise environments that restrict outbound traffic. The server listens on port 8181.
    If the OPA container crashes with a SIGSEGV segmentation fault in automaxprocs, you might be running on an emulated architecture such as an ARM-based Mac with a Kind cluster. Try using the statically linked image variant instead, such as openpolicyagent/opa:0.70.0-static.
  2. Create a GatewayClass that references the EnterpriseKgatewayParameters resource.
    kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: enterprise-kgateway-opa
spec:
  controllerName: solo.io/enterprise-kgateway
  parametersRef:
    group: enterprisekgateway.solo.io
    kind: EnterpriseKgatewayParameters
    name: opa-sidecar-params
    namespace: kgateway-system
EOF
  3. Create a Gateway that uses the new GatewayClass, and an HTTPRoute to expose the httpbin app through the new Gateway.
    kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: http-opa
  namespace: kgateway-system
spec:
  gatewayClassName: enterprise-kgateway-opa
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-opa
  namespace: httpbin
spec:
  parentRefs:
    - name: http-opa
      namespace: kgateway-system
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF
  4. Verify that the ext-auth service for the new GatewayClass is healthy with 2 containers ready. The second container is the OPA sidecar.
    kubectl get pods -n kgateway-system -l app.kubernetes.io/instance=ext-auth-service-enterprise-kgateway-opa
    Example output:
    NAME                                                           READY   STATUS    RESTARTS   AGE
ext-auth-service-enterprise-kgateway-opa-6546584c8d-w9s4l   2/2     Running   0          97s
Create the OPA AuthConfig

Create the Solo Enterprise for kgateway resources to enforce the OPA policy through the sidecar.
  1. Create an AuthConfig resource that uses opaServerAuth to connect to the OPA sidecar running in the same pod.
    kubectl apply -f - <<EOF
apiVersion: extauth.solo.io/v1
kind: AuthConfig
metadata:
  name: opa-sidecar
  namespace: httpbin
spec:
  configs:
  - name: opa
    opaServerAuth:
      serverAddr: http://localhost:8181
      package: test
      ruleName: result
EOF
    Review the following table to understand this configuration.
    SettingDescription
    nameA name for this AuthConfig entry.
    opaServerAuthConfigure the OPA server authentication details. Use this setting to connect to an OPA server sidecar or an external OPA server.
    serverAddrThe address of the OPA server. Because the OPA server runs as a sidecar in the same pod, the two containers share a network namespace and can communicate over localhost.
    packageThe package name in your Rego policy bundle that you want to enforce. This example uses the test package from the config map policy.
    ruleNameThe Rego rule to evaluate for the authorization decision. This example queries the result rule, which returns a structured object with allow (boolean) and http_status (integer) fields. For more information about rule names, see the OPA Data API docs.
  2. Create an EnterpriseKgatewayTrafficPolicy resource that refers to the AuthConfig that you created. The following policy applies external auth to all routes that the http-opa Gateway serves.
    kubectl apply -f - <<EOF
apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayTrafficPolicy
metadata:
  name: opa-sidecar
  namespace: kgateway-system
spec:
  targetRefs:
    - name: http-opa
      group: gateway.networking.k8s.io
      kind: Gateway
  entExtAuth:
    authConfigRef:
      name: opa-sidecar
      namespace: httpbin
EOF
  3. Verify that the AuthConfig is ACCEPTED.
    kubectl get authconfig opa-sidecar -n httpbin -o yaml
Verify the OPA policy

  1. Get the address of the http-opa Gateway.
    export INGRESS_GW_ADDRESS=$(kubectl get svc -n kgateway-system http-opa -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS
    kubectl port-forward deployment/http-opa -n kgateway-system 8080:8080
  2. Send a request to the httpbin app without the allow header. Verify that your request is denied and that you get back a 403 HTTP response code.
    curl -v http://$INGRESS_GW_ADDRESS:8080/get -H "host: www.example.com:8080"
    curl -v localhost:8080/get -H "host: www.example.com"
    Example output:
    < HTTP/1.1 403 Forbidden
< server: envoy
< content-length: 0
  3. Send another request, this time with the allow: true header. Verify that the request succeeds and that you get back a 200 HTTP response code.
    curl -v http://$INGRESS_GW_ADDRESS:8080/get -H "host: www.example.com:8080" -H "allow: true"
    curl -v localhost:8080/get -H "host: www.example.com" -H "allow: true"
    Example output:
    < HTTP/1.1 200 OK
< access-control-allow-credentials: true
< access-control-allow-origin: *
< content-type: application/json; encoding=utf-8
< x-envoy-upstream-service-time: 1
< server: envoy
Cleanup

You can optionally remove the resources that you set up as part of this guide.
kubectl delete authconfig opa-sidecar -n httpbin
kubectl delete EnterpriseKgatewayTrafficPolicy opa-sidecar -n kgateway-system
kubectl delete httproute httpbin-opa -n httpbin
kubectl delete gateway http-opa -n kgateway-system
kubectl delete configmap opa-policy -n kgateway-system
kubectl delete gatewayclass enterprise-kgateway-opa
kubectl delete EnterpriseKgatewayParameters opa-sidecar-params -n kgateway-system
rm policy.rego
Bring your own OPA server
Was this page helpful?