Navigation :

OPA server as a sidecar

Enforce Open Policy Agent (OPA) policies for more fine-grained access control.

You can enable an OPA server to run as a sidecar to the Gloo Platform external auth service. Then, you can administer an OPA server for extended use cases such as bundling. With bundling, your Rego rules can live as a signed bundle in an external, central location, such as an AWS S3 bucket to meet your internal security requirements. This sidecar approach increases the resources needed in the external auth server pod, but works better at scale and provides more OPA-native support for teams familiar with administering an OPA server.

Other OPA options:

Load Rego rules as Kubernetes config maps to the Gloo external auth service OPA module.
Instead of deploying a sidecar, you can bring your own OPA server.

Support for running the OPA server sidecar is a beta feature available in Gloo Platform 2.4.1 and later. Beta features might change and are not supported for production. For more information, see Gloo feature maturity.

If you import or export resources across workspaces, your policies might not apply. For more information, see Import and export policies.

About the OPA sidecar

Gloo Platform can deploy an OPA sidecar server for you. After, you are responsible for administering the server per OPA best practices.

The following diagram and the steps in the rest of this guide show how you can set up an OPA server sidecar.

Figure: Architecture for deploying an OPA server sidecar.

A user sends a request that the ingress gateway receives. The request matches a route that is protected by an external auth policy that uses OPA.
The ingress gateway sends the request to the external auth service for an authorization decision.
The external auth service passes the request through to the OPA server sidecar to make an authorization decision.
The OPA server sidecar loads the OPA config of Rego rules from a bundle in a cloud provider. The OPA server uses these Rego rules to make an authorization decision on the request. You can provide the OPA config via a YAML file during the Helm installation, or subsequently in a Kubernetes config map. Note that the request does not trigger loading the rules. You must restart the OPA server each time that you update the OPA config.
The OPA server returns the authorization decision to the external auth service, which returns the authorization decision to the ingress gateway.
The ingress gateway handles the request per the authorization decision.
- If unauthorized, the ingress gateway denies the request.
- If authorized, the ingress gateway forwards the request to the destination workload.

Before you begin

This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.

Complete the multicluster getting started guide to set up the following testing environment.
- Three clusters along with environment variables for the clusters and their Kubernetes contexts.
- The Gloo Platform CLI, meshctl, along with other CLI tools such as kubectl and istioctl.
- The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
- Istio installed in the workload clusters.
- A simple Gloo workspace setup.
Install Bookinfo and other sample apps.
Make sure that the external auth service is installed and running. If not, install the external auth service in your single or multicluster environment.
```
kubectl get pods -A -l app=ext-auth-service --context ${REMOTE_CONTEXT1}
```
Download the opa CLI tool.

Bundle Rego rules

Prepare OPA configuration for the OPA server by creating, bundling, and referring to a Rego policy with the rules you want to enforce.

Create a Rego policy file in a rego directory with the rules you want to enforce with OPA. The following policy allows GET and POST HTTP requests and denies requests to the /status endpoint.

mkdir rego
cat <<EOF > rego/policy.rego
package httpbin
   
import input.http_request
   
# deny requests by default
default allowed = false
   
# set allowed to true if no error message
allowed {
    not body
}
   
# return result and error message
allow["allowed"] = allowed
allow["body"] = body

# main policy logic, with error message per rule
body = "HTTP verb is not allowed" { not http_verb_allowed }
else = "Path is not allowed" { not path_allowed }

# allow only GET and POST requests
http_verb_allowed {
   {"GET", "POST"}[_] == http_request.method
}

# deny requests to /status endpoint
path_allowed {
   not startswith(http_request.path, "/status")
}
EOF

Use the opa CLI to bundle your Rego rules. The output of the command is a bundle.tar.gz compressed file in your current directory. For more information, see the OPA docs.

Signed policies: To make sure that your Rego policies come from a trusted source, you can add a signature to your bundle. For more information, see the OPA docs.
```
opa build -b rego/
```
Store your bundle in a supported cloud provider. For options and steps, see the OPA implementation docs.

Create a YAML file with your OPA config. For more information, see the OPA implementation docs.

cat > config.yaml - <<EOF
services:
  gcs:
    url: https://storage.googleapis.com/storage/v1/b/test-opa-bundles
bundles:
  bundle:
    service: gcs
    resource: 'bundle.tar.gz?alt=media'
EOF

Review the following table to understand this configuration.

Setting	Description
`services`	Provide the details of the cloud service where the bundle is located. In this example, the bundle is in a Google Cloud Storage (`gcs`) bucket at a test URL. The test URL is public, but you can also set up secure access with the `credentials` settings.
`bundles`	Provide the details about the particular bundle that you want to use. In this example, the bundle is the `bundle.tar.gz` bundle that you previously created.
Other settings	If you use other features, you can configure those settings in the config map. Common settings include signed bundles and returning decision logs. For more information, see the OPA implementation docs.

Upgrade the external auth server to run the OPA sidecar

Deploy the OPA server sidecar by upgrading the external auth service in your existing Gloo Platform installation. You can also set up the sidecar when you first install Gloo Platform.

The following steps upgrade an existing Helm release to add the OPA server as a sidecar to the external auth service. The steps do not upgrade the Gloo Platform management server or agent versions or otherwise change the components.

Check the Helm releases in your cluster. Depending on your installation method, you either have only a main installation release (such as gloo-platform), or a main installation and a separate add-ons release (such as gloo-agent-addons), in addition to your CRDs release.
```
helm ls -A --kube-context ${REMOTE_CONTEXT1}
```
Get your current installation values.
- If you have only one release for your installation, get those values. Note that if you migrated from the legacy Helm charts, your Helm release might be named gloo-mgmt or gloo-mesh-enterprise instead.
```
helm get values gloo-platform -n gloo-mesh --kube-context ${REMOTE_CONTEXT1} > gloo-gateway-single.yaml
open gloo-gateway-single.yaml
```
- If you have a separate add-ons release, get those values.
```
helm get values gloo-agent-addons -n gloo-mesh-addons --kube-context ${REMOTE_CONTEXT1} > gloo-agent-addons.yaml
open gloo-agent-addons.yaml
```
Delete the first line that contains USER-SUPPLIED VALUES:, and save the file.
Add or edit the following settings to deploy the OPA server sidecar as part of the external auth service. For other settings, see the opaServer settings in the Helm reference docs.
```
extAuthService:
  enabled: true
  extAuth:
    opaServer:
      enabled: true
      configYaml: ""
```
Upgrade your Helm release with the OPA server sidecar. Be sure to include the OPA config file that you previously created.
- If you have only one release for your installation, upgrade the gloo-platform release. Note that if you migrated from the legacy Helm charts, your Helm release might be named gloo-mgmt or gloo-mesh-enterprise instead.
```
helm upgrade gloo-platform gloo-platform/gloo-platform --kube-context ${REMOTE_CONTEXT1} \
   --namespace gloo-mesh \
   -f gloo-gateway-single.yaml \
   --set-file configYaml=config.yaml \
   --version $GLOO_VERSION
```
- If you have a separate add-ons release, upgrade the gloo-agent-addons release.
```
helm upgrade gloo-agent-addons gloo-platform/gloo-platform --kube-context ${REMOTE_CONTEXT1} \
   --namespace gloo-mesh-addons \
   -f gloo-agent-addons.yaml \
   --set-file configYaml=config.yaml \
   --version $GLOO_VERSION
```
Note: This Helm upgrade loads the initial OPA config for the OPA server to test an example policy. Later, if you want to update the OPA config, see Update OPA config.
Verify that the external auth service is healthy, with 2 containers ready.
```
kubectl get po -n gloo-mesh-addons -l app=ext-auth-service --context ${REMOTE_CONTEXT1}
```
```
NAME                                READY   STATUS    RESTARTS   AGE
ext-auth-service-6546584c8d-w9s4l   2/2     Running   0          97s
```
If the OPA sidecar is not healthy, the external auth service cannot enter a healthy running state, either. Continue to the next step to check the OPA server.

Check that the OPA server is healthy and accepted the config. Note the following example command pipes the output to jq for readability.

Confirm that the OPA server loaded the bundle that you referred to in the config map.
Confirm the health checks return a 200 response status.
If the OPA server is not healthy, try the OPA troubleshooting docs. Common errors include misconfiguration such as the wrong credentials to download a bundle or the wrong URL to the storage bucket.

kubectl logs -n gloo-mesh-addons deploy/ext-auth-service -c opa-auth --context ${REMOTE_CONTEXT1} | jq

{
  "level": "info",
  "msg": "Bundle loaded and activated successfully. Etag updated to CMzq8eO/p4EDEAE=.",
  "name": "httpbin",
  "plugin": "bundle",
  "time": "2023-09-20T15:54:47Z"
}
{
  "client_addr": "@",
  "level": "info",
  "msg": "Received request.",
  "req_id": 2,
  "req_method": "GET",
  "req_path": "/health",
  "time": "2023-09-20T15:54:48Z"
}
{
  "client_addr": "@",
  "level": "info",
  "msg": "Sent response.",
  "req_id": 2,
  "req_method": "GET",
  "req_path": "/health",
  "resp_bytes": 3,
  "resp_duration": 0.499848,
  "resp_status": 200,
  "time": "2023-09-20T15:54:48Z"
}

Create the OPA external auth policy

Create the Gloo external auth resources to enforce the OPA policy.

Create an external auth server to use for your policy.

kubectl apply --context ${REMOTE_CONTEXT1} -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: ExtAuthServer
metadata:
  name: opa-ext-auth-server
  namespace: httpbin
spec:
  destinationServer:
    port:
      number: 8083
    ref:
      cluster: $REMOTE_CLUSTER1
      name: ext-auth-service
      namespace: gloo-mesh-addons
EOF

Create an external auth policy that uses the OPA config map.

When you create the policy with a destination selector, only Kubernetes services can be specified in the applyToDestination section. Gloo virtual destinations or Gloo external services are not supported.

kubectl apply --context ${REMOTE_CONTEXT1} -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: ExtAuthPolicy
metadata:
  name: opa-sidecar
  namespace: httpbin
spec:
  applyToDestinations:
  - selector:
      labels:
        app: httpbin
  config:
    server:
      name: opa-ext-auth-server
      namespace: httpbin
    glooAuth:
      configs:
      - opaServerAuth:
          package: httpbin
          ruleName: allow/allowed
EOF

Review the following table to understand this configuration. For more information, see the API reference.

Setting	Description
`applyToDestinations`	Configure which destinations to apply the policy to, by using labels. Destinations can be a Kubernetes service, VirtualDestination, or ExternalService. If you do not specify any destinations or routes, the policy applies to all destinations in the workspace by default. If you do not specify any destinations but you do specify a route, the policy applies to the route but to no destinations.
`server`	The external auth server to use for the policy.
`opaServerAuth`	Configure the OPA server sidecar authentication details.
`package`	Refer to the package in the Rego bundle that you set up in the config map earlier, such as `httpbin` from the `policy.rego` file.
`ruleName`	Select the Rego rules within the bundle that you want to enforce for this OPA external auth policy. From the `policy.rego` file, you set the `allow` input document and only the `allowed` decision for that document, `allow/allowed`. For more information about rule names, see the OPA Data API docs.

Confirm that the external auth policy's state is ACCEPTED.

kubectl get -n httpbin ExtAuthPolicy opa-sidecar -o yaml --context ${REMOTE_CONTEXT1}

Verify the OPA external auth policy

Verify that the Rego rules are evaluated by the OPA server and enforced by the external auth service.

Send an allowed request to the httpbin app, such as a GET bytes request. You get a 200 response.
- Kubernetes 1.22 or earlier
- Kubernetes 1.23 or later
Create a temporary curl pod in the bookinfo namespace, so that you can test the app setup. You can also use this method in Kubernetes 1.23 or later, but an ephemeral container might be simpler, as shown in the other tab.
1. Create the curl pod.
  kubectl run -it -n httpbin --context ${REMOTE_CONTEXT1} curl \ --image=curlimages/curl:7.73.0 --rm -- sh
2. Send a request to the httpbin app.
  curl -H "X-httpbin: true" -v http://httpbin:8000/bytes/5
Use the kubectl debug command to create an ephemeral curl container in the deployment. This way, the curl container inherits any permissions from the app that you want to test. If you don't run Kubernetes 1.23 or later, you can deploy a separate curl pod or manually add the curl container as shown in the other tab.
```
kubectl --context ${REMOTE_CONTEXT1} -n httpbin debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=httpbin -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -H "X-httpbin: true" -v http://httpbin:8000/bytes/5
```
If the output has an error about EphemeralContainers, see Ephemeral containers don’t work when testing Bookinfo.
Send the request again along a path that is not allowed by the OPA policy, such as /status. Now, the request is blocked with a 404 Not Found response because the endpoint cannot be accessed.
- Kubernetes 1.22 or earlier
- Kubernetes 1.23 or later
Create a temporary curl pod in the bookinfo namespace, so that you can test the app setup. You can also use this method in Kubernetes 1.23 or later, but an ephemeral container might be simpler, as shown in the other tab.
1. Create the curl pod.
  kubectl run -it -n httpbin --context ${REMOTE_CONTEXT1} curl \ --image=curlimages/curl:7.73.0 --rm -- sh
2. Send a request to the httpbin app.
  curl -H "X-httpbin: true" -v http://httpbin:8000/status
Use the kubectl debug command to create an ephemeral curl container in the deployment. This way, the curl container inherits any permissions from the app that you want to test. If you don't run Kubernetes 1.23 or later, you can deploy a separate curl pod or manually add the curl container as shown in the other tab.
```
kubectl --context ${REMOTE_CONTEXT1} -n httpbin debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=httpbin -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -H "X-httpbin: true" -v http://httpbin:8000/status
```
This time, send a request to an allowed endpoint but with an HTTP method that is not allowed by the OPA policy, such as PUT. The request is blocked with a 405 Method Not Allowed response.
- Kubernetes 1.22 or earlier
- Kubernetes 1.23 or later
Create a temporary curl pod in the bookinfo namespace, so that you can test the app setup. You can also use this method in Kubernetes 1.23 or later, but an ephemeral container might be simpler, as shown in the other tab.
1. Create the curl pod.
  kubectl run -it -n httpbin --context ${REMOTE_CONTEXT1} curl \ --image=curlimages/curl:7.73.0 --rm -- sh
2. Send a request to the httpbin app.
  curl -H "X-httpbin: true" -X PUT -v http://httpbin:8000/status
Use the kubectl debug command to create an ephemeral curl container in the deployment. This way, the curl container inherits any permissions from the app that you want to test. If you don't run Kubernetes 1.23 or later, you can deploy a separate curl pod or manually add the curl container as shown in the other tab.
```
kubectl --context ${REMOTE_CONTEXT1} -n httpbin debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=httpbin -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -H "X-httpbin: true" -X PUT -v http://httpbin:8000/status
```

Cleanup

Optionally, you can remove the external auth resources and the OPA server sidecar.

Delete the resources that you created.

kubectl -n gloo-mesh-addons delete ConfigMap opa-config --context ${REMOTE_CONTEXT1}
kubectl -n httpbin delete ExtAuthServer opa-ext-auth-server --context ${REMOTE_CONTEXT1}
kubectl -n httpbin delete ExtAuthPolicy opa-server --context ${REMOTE_CONTEXT1}

Repeat the Helm upgrade steps, disabling the OPA server sidecar in the external auth service in the Helm values file.
```
extAuthService:
  enabled: true
  extAuth:
    opaServer:
      enabled: false
```

Update OPA config

To update OPA config after initially deploying the OPA server sidecar, choose from the following options.

Upgrade your Helm installation with the new OPA config.
Create a Kubernetes config map with the new OPA config and restart the OPA server, as show in the following steps.

Steps for creating and updating config maps:

Follow Steps 1 - 3 of Bundle Rego rules to create, bundle, and store your Rego rules in a cloud provider.

Create a Kubernetes config map that refers to your bundle.

kubectl apply --context ${REMOTE_CONTEXT1} -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-config
  namespace: gloo-mesh-addons
data:
  config.yaml: |
    services:
      gcs:
        url: https://storage.googleapis.com/storage/v1/b/test-opa-bundles
    bundles:
      bundle:
        service: gcs
        resource: 'bundle.tar.gz?alt=media'
EOF

Review the following table to understand this configuration.

Setting	Description
`namespace`	Create the config map in the same namespace as the external auth server, such as `gloo-mesh-addons`.
`config.yaml`	Enter the bundle configuration details that you created as part of uploading your bundle to the cloud provider. For more information, see the OPA implementation docs.
`services`	Provide the details of the cloud service where the bundle is located. In this example, the bundle is in a Google Cloud Storage (`gcs`) bucket at a test URL. The test URL is public, but you can also set up secure access with the `credentials` settings.
`bundles`	Provide the details about the particular bundle that you want to use. In this example, the bundle is the `bundle.tar.gz` bundle that you previously created.
Other settings	If you use other features, you can configure those settings in the config map. Common settings include signed bundles and returning decision logs. For more information, see the OPA implementation docs.

For your OPA config changes to take effect, restart the external auth service with the OPA server sidecar. You must restart the OPA server each time that you update the OPA config. Optionally, you can create the config map before enabling the OPA server sidecar and refer to the config map during the Helm installation. This way, the first time the OPA server sidecar comes up, it has the OPA config already, reducing the number of times you might have to restart the service.
```
kubectl rollout restart deployment/ext-auth-service -n gloo-mesh-addons --context ${REMOTE_CONTEXT1}
```

Optional: Check that the OPA server is healthy and accepted the config. Note the following example command pipes the output to jq for readability.

Confirm that the OPA server loaded the bundle that you referred to in the config map.
Confirm the health checks return a 200 response status.

kubectl logs -n gloo-mesh-addons deploy/ext-auth-service -c opa-auth --context ${REMOTE_CONTEXT1} | jq

{
  "level": "info",
  "msg": "Bundle loaded and activated successfully. Etag updated to CMzq8eO/p4EDEAE=.",
  "name": "httpbin",
  "plugin": "bundle",
  "time": "2023-09-20T15:54:47Z"
}
{
  "client_addr": "@",
  "level": "info",
  "msg": "Received request.",
  "req_id": 2,
  "req_method": "GET",
  "req_path": "/health",
  "time": "2023-09-20T15:54:48Z"
}
{
  "client_addr": "@",
  "level": "info",
  "msg": "Sent response.",
  "req_id": 2,
  "req_method": "GET",
  "req_path": "/health",
  "resp_bytes": 3,
  "resp_duration": 0.499848,
  "resp_status": 200,
  "time": "2023-09-20T15:54:48Z"
}