OPA server as a sidecar

You can enable an OPA server to run as a sidecar to the Gloo external auth service. Then, you can administer an OPA server for extended use cases such as bundling. With bundling, your Rego rules can live as a signed bundle in an external, central location, such as an AWS S3 bucket to meet your internal security requirements. This sidecar approach increases the resources needed in the external auth server pod, but works better at scale and provides more OPA-native support for teams familiar with administering an OPA server. You also get the OPA-Envoy plugin API as part of the Gloo external auth service.

Other OPA options:

• Load Rego rules as Kubernetes config maps to the Gloo external auth service OPA module.
• Instead of deploying a sidecar, you can bring your own OPA server.

About the OPA sidecar link

Gloo Mesh Gateway can deploy an OPA sidecar server for you. After, you are responsible for administering the server per OPA best practices.

The following diagram and the steps in the rest of this guide show how you can set up an OPA server sidecar.

1. A user sends a request that the ingress gateway receives. The request matches a route that is protected by an external auth policy that uses OPA.
2. The ingress gateway sends the request to the external auth service for an authorization decision.
3. The external auth service passes the request through to the OPA server sidecar to make an authorization decision.
4. The OPA server sidecar loads the OPA config of Rego rules from a bundle in a cloud provider. The OPA server uses these Rego rules to make an authorization decision on the request. You can provide the OPA config via a YAML file during the Helm installation, or subsequently in a Kubernetes config map. Note that the request does not trigger loading the rules. You must restart the OPA server each time that you update the OPA config.
5. The OPA server returns the authorization decision to the external auth service, which returns the authorization decision to the ingress gateway.
6. The ingress gateway handles the request per the authorization decision.
   • If unauthorized, the ingress gateway denies the request.
   • If authorized, the ingress gateway forwards the request to the destination workload.

Clear route cache link

The Envoy proxy in your sidecar instance keeps a route cache in memory of precomputed routing decisions to help speed up performance. When the sidecar gets a request, it can check the route cache to decide where to send the request. When the route cache is cleared, Gloo recomputes the routing rules. This way, any old and potentially conflicting data from the initial request is cleared.

Before you begin link

1. Set up Gloo Mesh Gateway in a single cluster.
2. Install Bookinfo and other sample apps.

3. Configure an HTTP listener on your gateway and set up basic routing for the sample apps.

4. Make sure that the external auth service is installed and running. If not, install the external auth service in your Gloo environment.

     kubectl get pods  -A -l app=ext-auth-service
     

5. Download the opa CLI tool.

6. Get the external address of your ingress gateway. The steps vary depending on the type of load balancer that backs the ingress gateway.

• LoadBalancer IP address:

    export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  echo $INGRESS_GW_IP
    

• LoadBalancer hostname:

    export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
  echo $INGRESS_GW_IP
    

Note: Depending on your environment, you might see <pending> instead of an external IP address. For example, if you are testing locally in kind or minikube, or if you have insufficent permissions in your cloud platform, you can instead port-forward the service port of the ingress gateway:

  kubectl -n gloo-mesh-gateways port-forward deploy/istio-ingressgateway-1-22 8081
  

Bundle Rego rules link

Prepare OPA configuration for the OPA server by creating, bundling, and referring to a Rego policy with the rules you want to enforce.

1. Create a Rego policy file in a rego directory with the rules you want to enforce with OPA. The following policy allows GET and POST HTTP requests and denies requests to the /status endpoint.

     mkdir rego
   cat <<EOF > rego/policy.rego
   package httpbin
   
   import input.http_request
   
   # deny requests by default
   default allowed = false
   
   # set allowed to true if no error message
   allowed {
       not body
   }
   
   # return result and error message
   allow["allowed"] = allowed
   allow["body"] = body
   
   # main policy logic, with error message per rule
   body = "HTTP verb is not allowed" { not http_verb_allowed }
   else = "Path is not allowed" { not path_allowed }
   
   # allow only GET and POST requests
   http_verb_allowed {
      {"GET", "POST"}[_] == http_request.method
   }
   
   # deny requests to /status endpoint
   path_allowed {
      not startswith(http_request.path, "/status")
   }
   EOF
     

2. Create another Rego policy. The following policy uses metadata that is enabled by the OPA-Envoy plugin API. This example shows how you can use OPA to evaluate and transform responses based on request metadata.

   • Check requests for an api-key header, with valid API keys either set to the test value of test-apikey or located in data.apikeys. The API keys in data.apikeys are loaded as part of the data bundle.
   • If the request is authorized, the api-key header is removed.
   • If the request is not authorized, the body is set to Unauthorized Request and returned to the client with a 403 status code, along with a reject-reason response header.
   • In both cases, the x-response-header is added.

     cat <<EOF > rego/apikey_policies.rego
   package apikey_policies
   
   import future.keywords.if
   
   default allow := false
   
   allow {
       api_key_allowed
   }
   
   api_key_allowed {
       api_key == "test-apikey"
   }
   api_key_allowed {
       has_key(data.apikeys, api_key)
   }
   
   api_key := input.http_request.headers["api-key"]
   
   http_status := 200 {
       allow
   }
   
   http_status := 403 {
       not allow
   }
   
   body := "Unauthorized Request" {
       http_status == 403
   }
   
   headers["x-ext-auth-allow"] := "yes" if api_key_allowed
   headers["x-validated-by"] := "security-checkpoint" if api_key_allowed
   
   request_headers_to_remove := ["api-key"]
   
   response_headers_to_add["x-response-header"] := "for-client-only"
   response_headers_to_add["reject-reason"] := "unauthorized" if not allow
   
   # Helper function to check if a dictionary has a key
   has_key(dict, k) {
       dict[k]
   }
   EOF
     

3. Use the opa CLI to bundle your Rego rules. The output of the command is a bundle.tar.gz compressed file in your current directory. For more information, see the OPA docs.

   check_circle

   Signed policies: To make sure that your Rego policies come from a trusted source, you can add a signature to your bundle. For more information, see the OPA docs.

     opa build -b rego/
     

4. Store your bundle in a supported cloud provider. For options and steps, see the OPA implementation docs.

   Example command for Google Cloud: Make sure to update the ${BUCKET_NAME} with the name of the gcs bucket.

     gsutil cp bundle.tar.gz gs://${BUCKET_NAME}/
     

5. Create a YAML file with your OPA config. For more information, see the OPA implementation docs.

     cat > config.yaml - <<EOF
   services:
     gcs:
       url: https://storage.googleapis.com/storage/v1/b/${BUCKET_NAME}/o
   bundles:
     bundle:
       service: gcs
       resource: 'bundle.tar.gz?alt=media'
   EOF
     

   Review the following table to understand this configuration.

   Setting	Description
   services	Provide the details of the cloud service where the bundle is located. In this example, the bundle is in a Google Cloud Storage (gcs) bucket. Make sure to update the ${BUCKET_NAME} with the name of the gcs bucket. For quick testing, you can use a bucket with a public URL. Or, you can also set up secure access with the credentials settings.
   bundles	Provide the details about the particular bundle that you want to use. In this example, the bundle is the bundle.tar.gz bundle that you previously created.
   Other settings	If you use other features, you can configure those settings in the config map. Common settings include signed bundles and returning decision logs. For more information, see the OPA implementation docs.

Upgrade the external auth server to run the OPA sidecar link

Deploy the OPA server sidecar by upgrading the external auth service in your existing Gloo Mesh Gateway installation. You can also set up the sidecar when you first install Gloo Mesh Gateway.

1. Check the Helm releases in your cluster. Depending on your installation method, you either have only a main installation release (such as gloo-platform), or a main installation and a separate add-ons release (such as gloo-agent-addons), in addition to your CRDs release.

     helm ls -A 
     

2. Get your current installation values.

   • If you have only one release for your installation, get those values. Note that your Helm release might have a different name.

       helm get values gloo-platform -n gloo-mesh -o yaml  > gloo-single.yaml
     open gloo-single.yaml
       

   • If you have a separate add-ons release, get those values.

       helm get values gloo-agent-addons -n gloo-mesh-addons -o yaml  > gloo-agent-addons.yaml
     open gloo-agent-addons.yaml
       

3. Add or edit the following settings to deploy the OPA server sidecar as part of the external auth service. For other settings, see the opaServer settings in the Helm reference docs.

     
   extAuthService:
     enabled: true
     extAuth:
       opaServer:
         enabled: true
         configYaml: ""
     

4. Upgrade your Helm release with the OPA server sidecar. Be sure to include the OPA config file that you previously created.

   • If you have only one release for your installation, upgrade the gloo-platform release. Note that your Helm release might have a different name.

       helm upgrade gloo-platform gloo-platform/gloo-platform  \
        --namespace gloo-mesh \
        -f gloo-single.yaml \
        --set-file configYaml=config.yaml \
        --version $GLOO_VERSION
       

   • If you have a separate add-ons release, upgrade the gloo-agent-addons release.

       helm upgrade gloo-agent-addons gloo-platform/gloo-platform  \
        --namespace gloo-mesh-addons \
        -f gloo-agent-addons.yaml \
        --set-file configYaml=config.yaml \
        --version $GLOO_VERSION
       

   info

   This Helm upgrade loads the initial OPA config for the OPA server to test an example policy. Later, if you want to update the OPA config, see Update OPA config.

5. Verify that the external auth service is healthy, with 2 containers ready.

     kubectl get po -n gloo-mesh-addons -l app=ext-auth-service 
     

     NAME                                READY   STATUS    RESTARTS   AGE
   ext-auth-service-6546584c8d-w9s4l   2/2     Running   0          97s
     

   warning

   If the OPA sidecar is not healthy, the external auth service cannot enter a healthy running state, either. Continue to the next step to check the OPA server.

6. Check that the OPA server is healthy and accepted the config. Note the following example command pipes the output to jq for readability.

   • Confirm that the OPA server loaded the bundle that you referred to in the config map.
   • Confirm the health checks return a 200 response status.
   • If the OPA server is not healthy, try the OPA troubleshooting docs. Common errors include misconfiguration such as the wrong credentials to download a bundle or the wrong URL to the storage bucket.

     kubectl logs -n gloo-mesh-addons deploy/ext-auth-service -c opa-auth  | jq
     

      {
        "level": "info",
        "msg": "Bundle loaded and activated successfully. Etag updated to CMzq8eO/p4EDEAE=.",
        "name": "httpbin",
        "plugin": "bundle",
        "time": "2023-09-20T15:54:47Z"
      }
      {
        "client_addr": "@",
        "level": "info",
        "msg": "Received request.",
        "req_id": 2,
        "req_method": "GET",
        "req_path": "/health",
        "time": "2023-09-20T15:54:48Z"
      }
      {
        "client_addr": "@",
        "level": "info",
        "msg": "Sent response.",
        "req_id": 2,
        "req_method": "GET",
        "req_path": "/health",
        "resp_bytes": 3,
        "resp_duration": 0.499848,
        "resp_status": 200,
        "time": "2023-09-20T15:54:48Z"
      }
      

Create the OPA external auth policy link

Create the Gloo external auth resources to enforce the OPA policy.

1. Create an external auth server to use for your policy.

     kubectl apply  -f - <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: ExtAuthServer
   metadata:
     name: opa-ext-auth-server
     namespace: httpbin
   spec:
     destinationServer:
       port:
         number: 8083
       ref:
         cluster: $CLUSTER_NAME
         name: ext-auth-service
         namespace: gloo-mesh-addons
   EOF
     

2. Create an external auth policy that uses the OPA config map.

     kubectl apply --context ${REMOTE_CONTEXT1} -f - <<EOF
   apiVersion: security.policy.gloo.solo.io/v2
   kind: ExtAuthPolicy
   metadata:
     name: opa-sidecar
     namespace: httpbin
   spec:
     applyToRoutes:
     - routes:
         labels:
           route: httpbin
     config:
       server:
         name: opa-ext-auth-server
         namespace: httpbin
       glooAuth:
         configs:
         - opaServerAuth:
             package: httpbin
             ruleName: allow/allowed
         - opaServerAuth:
             package: apikey_policies
   EOF
     

   Review the following table to understand this configuration. For more information, see the API reference.

   Setting	Description
   applyToRoutes	Use labels to configure which routes to apply the policy to. This example label matches the app and route from the example route table that you apply separately. If omitted and you do not have another selector such as applyToDestinations, the policy applies to all routes in the workspace.
   server	The external auth server to use for the policy.
   opaServerAuth	Configure the OPA server sidecar authentication details.
   package	Refer to the package in the Rego bundle that you set up in the config map earlier, such as httpbin from the policy.rego file.
   ruleName	Select the Rego rules within the bundle that you want to enforce for this OPA external auth policy. From the policy.rego file, you set the allow input document and only the allowed decision for that document, allow/allowed. For the apikey_policies.rego file, you do not set a certain rule. This way, the response has all the objects in the policy, even ones that are not used. You can take this approach with short policies like this example. However, for more complex scenarios, consult the OPA primer to create objects that you can refer to in the external auth policy. For more information about rule names, see the OPA Data API docs.

Verify the OPA external auth policy link

Verify that the Rego rules are evaluated by the OPA server and enforced by the external auth service.

1. Confirm that the external auth policy’s state is ACCEPTED.

     kubectl get -n httpbin ExtAuthPolicy opa-sidecar -o yaml 
     

2. Send an allowed request to the httpbin app, such as a GET bytes request with the test-apikey API key.

   • HTTP:

       curl -vik http://$INGRESS_GW_IP:80/bytes/5 -H "host: www.example.com:80" -H "X-httpbin: true" -H "api-key: test-apikey"
       

   • HTTPS:

       curl -vik https://$INGRESS_GW_IP:443/bytes/5 -H "host: www.example.com:443" -H "X-httpbin: true" -H "api-key: test-apikey"
       

   • In the response, you get a successful 200 status code. Also notice that the api-key header is removed and the x-response-header is added, as specified in the Rego policy.

       *   Trying 3.130.169.185:80...
     * Connected to $INGRESS_GW_IP port 80 (#0)
     > GET /bytes/5 HTTP/1.1
     > Host: www.example.com:80
     > User-Agent: curl/8.1.2
     > Accept: */*
     > X-httpbin: true
     > api-key: test-apikey
     > 
     HTTP/1.1 200 OK
     access-control-allow-credentials: true
     access-control-allow-origin: *
     content-type: application/octet-stream
     date: Thu, 16 Nov 2023 15:57:15 GMT
     content-length: 5
     x-envoy-upstream-service-time: 2
     x-response-header: for-client-only
     server: istio-envoy
     
     * Connection #0 to host $INGRESS_GW_IP left intact
     D?d?L% 
       

3. Repeat the request, but remove the test-apikey API key. You get a 403 response because the request must meet all the Rego policy requirements to succeed.

   • HTTP:

       curl -vik http://$INGRESS_GW_IP:80/bytes/5 -H "host: www.example.com:80" -H "X-httpbin: true"
       

   • HTTPS:

       curl -vik https://$INGRESS_GW_IP:443/bytes/5 -H "host: www.example.com:443" -H "X-httpbin: true"
       

   Notice that the response includes the transformations that you specified in the Rego policy, particularly:

   • The x-response-header: for-client-only is added to the response.
   • The reject-reason: unauthorized is added to the response.
   • The body returned says Unauthorized Request.

     * Connected to $INGRESS_GW_IP port 80 (#0)
   > GET /bytes/5 HTTP/1.1
   > Host: www.example.com:80
   > User-Agent: curl/8.1.2
   > Accept: */*
   > X-httpbin: true
   
   HTTP/1.1 403 Forbidden
   reject-reason: unauthorized
   x-response-header: for-client-only
   content-length: 20
   content-type: text/plain
   date: Thu, 16 Nov 2023 15:36:05 GMT
   server: istio-envoy
   * Connection #0 to host $INGRESS_GW_IP left intact
   Unauthorized Request
     

4. Send the request again along a path that is not allowed by the OPA policy, such as /status. Now, the request is blocked with a 404 Not Found response because the endpoint cannot be accessed.

   • HTTP:

       curl -vik http://$INGRESS_GW_IP:80/status -H "host: www.example.com:80" -H "X-httpbin: true" -H "api-key: test-apikey"
       

   • HTTPS:

       curl -vik https://$INGRESS_GW_IP:443/status -H "host: www.example.com:443" -H "X-httpbin: true" -H "api-key: test-apikey"
       

5. This time, send a request to an allowed endpoint but with an HTTP method that is not allowed by the OPA policy, such as PUT. The request is blocked with a 405 Method Not Allowed response.

   • HTTP:

       curl -vik http://$INGRESS_GW_IP:80/bytes/5 -H "host: www.example.com:80" -H "X-httpbin: true" -X PUT -H "api-key: test-apikey"
       

   • HTTPS:

       curl -vik https://$INGRESS_GW_IP:443/bytes/5 -H "host: www.example.com:443" -H "X-httpbin: true" -X PUT -H "api-key: test-apikey"
       

Cleanup link

You can optionally remove the resources that you set up as part of this guide.

1. Delete the resources that you created.

     kubectl -n gloo-mesh-addons delete ConfigMap opa-config 
   kubectl -n httpbin delete ExtAuthServer opa-ext-auth-server 
   kubectl -n httpbin delete ExtAuthPolicy opa-server 
     

2. Repeat the Helm upgrade steps, disabling the OPA server sidecar in the external auth service in the Helm values file.

     
   extAuthService:
     enabled: true
     extAuth:
       opaServer:
         enabled: false
     

Update OPA config link

To update OPA config after initially deploying the OPA server sidecar, choose from the following options.

• Upgrade your Helm installation with the new OPA config.
• Create a Kubernetes config map with the new OPA config and restart the OPA server, as show in the following steps.

Steps for creating and updating config maps:

1. Follow Steps 1 - 3 of Bundle Rego rules to create, bundle, and store your Rego rules in a cloud provider.

2. Create a Kubernetes config map that refers to your bundle.

     kubectl apply  -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: opa-config
     namespace: gloo-mesh-addons
   data:
     config.yaml: |
       services:
         gcs:
           url: https://storage.googleapis.com/storage/v1/b/${BUCKET_NAME}/o
       bundles:
         bundle:
           service: gcs
           resource: 'bundle.tar.gz?alt=media'
   EOF
     

   Review the following table to understand this configuration.

   Setting	Description
   namespace	Create the config map in the same namespace as the external auth server, such as gloo-mesh-addons.
   config.yaml	Enter the bundle configuration details that you created as part of uploading your bundle to the cloud provider. For more information, see the OPA implementation docs.
   services	Provide the details of the cloud service where the bundle is located. In this example, the bundle is in a Google Cloud Storage (gcs) bucket. Make sure to update the ${BUCKET_NAME} with the name of the gcs bucket. For quick testing, you can use a bucket with a public URL. Or, you can also set up secure access with the credentials settings.
   bundles	Provide the details about the particular bundle that you want to use. In this example, the bundle is the bundle.tar.gz bundle that you previously created.
   Other settings	If you use other features, you can configure those settings in the config map. Common settings include signed bundles and returning decision logs. For more information, see the OPA implementation docs.

3. For your OPA config changes to take effect, restart the external auth service with the OPA server sidecar. You must restart the OPA server each time that you update the OPA config. Optionally, you can create the config map before enabling the OPA server sidecar and refer to the config map during the Helm installation. This way, the first time the OPA server sidecar comes up, it has the OPA config already, reducing the number of times you might have to restart the service.

     kubectl rollout restart deployment/ext-auth-service -n gloo-mesh-addons 
     

4. Optional: Check that the OPA server is healthy and accepted the config. Note the following example command pipes the output to jq for readability.

   • Confirm that the OPA server loaded the bundle that you referred to in the config map.
   • Confirm the health checks return a 200 response status.

     kubectl logs -n gloo-mesh-addons deploy/ext-auth-service -c opa-auth  | jq
     

      {
        "level": "info",
        "msg": "Bundle loaded and activated successfully. Etag updated to CMzq8eO/p4EDEAE=.",
        "name": "httpbin",
        "plugin": "bundle",
        "time": "2023-09-20T15:54:47Z"
      }
      {
        "client_addr": "@",
        "level": "info",
        "msg": "Received request.",
        "req_id": 2,
        "req_method": "GET",
        "req_path": "/health",
        "time": "2023-09-20T15:54:48Z"
      }
      {
        "client_addr": "@",
        "level": "info",
        "msg": "Sent response.",
        "req_id": 2,
        "req_method": "GET",
        "req_path": "/health",
        "resp_bytes": 3,
        "resp_duration": 0.499848,
        "resp_status": 200,
        "time": "2023-09-20T15:54:48Z"
      }