Bring your own OPA server

By administering your own OPA server, you can make use of extended OPA use cases. For example, your Rego rules can live as a signed bundle in an external, central location, such as an AWS S3 bucket to meet your internal security requirements. Bringing your own OPA server increases the administrative complexity, but works better at scale and provides more OPA-native support for teams familiar with administering an OPA server. It also lets you take advantage of your existing OPA server configuration and enterprise OPA license.

Other OPA options:

• Load Rego rules as Kubernetes config maps to the Gloo external auth service OPA module.
• Instead of bringing your own server, you can deploy an OPA server as a sidecar to the Gloo external auth service.

About bringing your own OPA server link

When you bring your own OPA server, you are responsible for setting up and administering the server per OPA best practices. You have a few setup options:

• Deploy the OPA server to the same cluster as your Gloo external auth service. This option is best suited for testing purposes or simpler, single-cluster use cases.
• Host the OPA server in a remote location that is accessible from your cluster. For this approach, you must create an external service to represent the OPA server and update the Gloo external auth server to refer to this external service. This option is best suited for existing, enterprise OPA deployments in complex, multicluster production scenarios.

The following diagram and the steps in the rest of this guide show how you can set up your own OPA server.

1. A user sends a request that the ingress gateway receives. The request matches a route that is protected by an external auth policy that uses OPA.
2. The ingress gateway sends the request to the external auth service for an authorization decision.
3. The external auth service passes the request through to the OPA server to make an authorization decision.
   • If the OPA server is within the cluster, the external auth service can refer to the OPA server by using its Kubernetes service address.
   • If the OPA server is outside the cluster, the external auth service refers to the OPA server by a reachable address, such as on the same private network.
4. The OPA server loads the OPA config of Rego rules from a bundle in a cloud provider. The OPA server uses these Rego rules to make an authorization decision on the request. You can provide the OPA config via a YAML file during the initial installation, or subsequently in a Kubernetes config map. Note that the request does not trigger loading the rules. You must restart the OPA server each time that you update the OPA config.
5. The OPA server returns the authorization decision to the external auth service, which returns the authorization decision to the ingress gateway.
6. The ingress gateway handles the request per the authorization decision.
   • If unauthorized, the ingress gateway denies the request.
   • If authorized, the ingress gateway forwards the request to the destination workload.

Before you begin link

1. Set up Gloo Mesh Gateway in a single cluster.
2. Install Bookinfo and other sample apps.

3. Configure an HTTP listener on your gateway and set up basic routing for the sample apps.

4. Make sure that the external auth service is installed and running. If not, install the external auth service in your Gloo environment.

     kubectl get pods  -A -l app=ext-auth-service
     

5. Download the opa CLI tool.

6. Get the external address of your ingress gateway. The steps vary depending on the type of load balancer that backs the ingress gateway.

  export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS
  

Note: Depending on your environment, you might see <pending> instead of an external IP address. For example, if you are testing locally in kind or minikube, or if you have insufficient permissions in your cloud platform, you can instead port-forward the service port of the ingress gateway:

  kubectl -n gloo-mesh-gateways port-forward deploy/istio-ingressgateway-1-22 8081
  

Bundle Rego rules link

Prepare OPA configuration for the OPA server by creating, bundling, and referring to a Rego policy with the rules you want to enforce.

1. Create a Rego policy file in a rego directory with the rules you want to enforce with OPA. The following policy allows GET and POST HTTP requests and denies requests to the /status endpoint.

     mkdir rego
   cat <<EOF > rego/policy.rego
   package httpbin
   
   import input.http_request
   
   # deny requests by default
   default allowed = false
   
   # set allowed to true if no error message
   allowed {
       not body
   }
   
   # return result and error message
   allow["allowed"] = allowed
   allow["body"] = body
   
   # main policy logic, with error message per rule
   body = "HTTP verb is not allowed" { not http_verb_allowed }
   else = "Path is not allowed" { not path_allowed }
   
   # allow only GET and POST requests
   http_verb_allowed {
      {"GET", "POST"}[_] == http_request.method
   }
   
   # deny requests to /status endpoint
   path_allowed {
      not startswith(http_request.path, "/status")
   }
   EOF
     

2. Use the opa CLI to bundle your Rego rules. The output of the command is a bundle.tar.gz compressed file in your current directory. For more information, see the OPA docs.

   check_circle

   Signed policies: To make sure that your Rego policies come from a trusted source, you can add a signature to your bundle. For more information, see the OPA docs.

     opa build -b rego/
     

3. Store your bundle in a supported cloud provider. For options and steps, see the OPA implementation docs.

4. Create a Kubernetes config map that refers to your bundle.

     kubectl apply  -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: opa-config
     namespace: gloo-mesh
   data:
     config.yaml: |
       services:
         gcs:
           url: https://storage.googleapis.com/storage/v1/b/test-opa-bundles
       bundles:
         bundle:
           service: gcs
           resource: 'bundle.tar.gz?alt=media'
   EOF
     

   Review the following table to understand this configuration.

   Setting	Description
   namespace	Create the config map in the same namespace as the OPA server, such as gloo-mesh.
   config.yaml	Enter the bundle configuration details that you created as part of uploading your bundle to the cloud provider. For more information, see the OPA implementation docs.
   services	Provide the details of the cloud service where the bundle is located. In this example, the bundle is in a Google Cloud Storage (gcs) bucket at a test URL. The test URL is public, but you can also set up secure access with the credentials settings.
   bundles	Provide the details about the particular bundle that you want to use. In this example, the bundle is the bundle.tar.gz bundle that you previously created.
   Other settings	If you use other features, you can configure those settings in the config map. Common settings include signed bundles and returning decision logs. For more information, see the OPA implementation docs.

Set up the OPA server link

Set up your OPA server as a remote server or as a deployment within the cluster.

1. Deploy the OPA server. For steps, you can follow the OPA deployment docs. Make sure to update the deployment with the OPA config map that you previously created. Later, if you want to update the OPA config, see Update OPA config.

   • Currently, the Gloo external auth service does not have a way to send auth details to a protected remote OPA server. Make sure to run the OPA server on a trusted network that your cluster has access to, such as a separate Kubernetes cluster in the same virtual private cloud (VPC) network as your cluster with enabled connectivity.
   • If your cluster limits egress, you might use an external service to add the remote OPA server to your cluster’s service mesh.

   Deploy the OPA server in your cluster, such as in the same gloo-mesh namespace as your external auth service.

2. Verify that the OPA server is running.

     kubectl get pods -A -l app=opa 
     

3. Get the service details, which you use later to refer to the OPA server in the external auth policy.

     kubectl get svc -A -l app=opa 
     

   Example output: Later, you use the OPA service address <service>.<namespace>:<port>, such as http://opa.gloo-mesh:8181.

     NAMESPACE          NAME   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
   gloo-mesh          opa    NodePort   10.xx.xx.xxx   <none>        8181:32427/TCP   15s
     

4. Confirm that the OPA server loaded the bundle that you referred to in the config map. Note that the following example command pipes the output to jq for readability.

     kubectl logs -n gloo-mesh deploy/opa  | jq
     

      {
        "level": "info",
        "msg": "Bundle loaded and activated successfully. Etag updated to CMzq8eO/p4EDEAE=.",
        "name": "httpbin",
        "plugin": "bundle",
        "time": "2023-09-20T15:54:47Z"
      }
      

Create the OPA external auth policy link

Create the Gloo external auth resources to enforce the OPA policy.

1. Create an external auth server to use for your policy.

     kubectl apply  -f - <<EOF
   apiVersion: admin.gloo.solo.io/v2
   kind: ExtAuthServer
   metadata:
     name: opa-ext-auth-server
     namespace: httpbin
   spec:
     destinationServer:
       port:
         number: 8083
       ref:
         cluster: $CLUSTER_NAME
         name: ext-auth-service
         namespace: gloo-mesh
   EOF
     

2. Create an external auth policy that uses the OPA config map.

     kubectl apply -f - <<EOF
   apiVersion: security.policy.gloo.solo.io/v2
   kind: ExtAuthPolicy
   metadata:
     name: opa-server
     namespace: httpbin
   spec:
     applyToRoutes:
     - route:
         labels:
           route: httpbin
     config:
       server:
         name: opa-ext-auth-server
         namespace: httpbin
       glooAuth:
         configs:
         - opaServerAuth:
             package: httpbin
             ruleName: allow/allowed
             serverAddr: http://opa.gloo-mesh:8181
   EOF
     

   Review the following table to understand this configuration. For more information, see the API reference.

   Setting	Description
   applyToRoutes	Use labels to configure which routes to apply the policy to. This example label matches the app and route from the example route table that you apply separately. If omitted and you do not have another selector such as applyToDestinations, the policy applies to all routes in the workspace.
   server	The external auth server to use for the policy.
   opaServerAuth	Configure the OPA server sidecar authentication details.
   package	Refer to the package in the Rego bundle that you set up in the config map earlier, such as httpbin from the policy.rego file.
   ruleName	Select the Rego rules within the bundle that you want to enforce for this OPA external auth policy. From the policy.rego file, you set the allow input document and only the allowed decision for that document, allow/allowed. For more information about rule names, see the OPA Data API docs.
   serverAddr	The reachable address of the OPA server that you previously retrieved when you deployed the OPA server.

3. Confirm that the external auth policy’s state is ACCEPTED.

     kubectl get -n httpbin ExtAuthPolicy opa-server -o yaml 
     

Verify the OPA external auth policy link

Verify that the Rego rules are evaluated by the OPA server and enforced by the external auth service.

1. Send an allowed request to the httpbin app, such as a GET bytes request. You get a 200 response.

   • HTTP:

       curl -vik --resolve www.example.com:80:${INGRESS_GW_IP} -H "X-httpbin: true" http://www.example.com:80/bytes/5
       

   • HTTPS:

       curl -vik --resolve www.example.com:443:${INGRESS_GW_IP} -H "X-httpbin: true" https://www.example.com:443/bytes/5
       

2. Send the request again along a path that is not allowed by the OPA policy, such as /status. Now, the request is blocked with a 404 Not Found response because the endpoint cannot be accessed.

   • HTTP:

       curl -vik --resolve www.example.com:80:${INGRESS_GW_IP} -H "X-httpbin: true" http://www.example.com:80/status
       

   • HTTPS:

       curl -vik --resolve www.example.com:443:${INGRESS_GW_IP} -H "X-httpbin: true" https://www.example.com:443/status
       

3. This time, send a request to an allowed endpoint but with an HTTP method that is not allowed by the OPA policy, such as PUT. The request is blocked with a 405 Method Not Allowed response.

   • HTTP:

       curl -vik --resolve www.example.com:80:${INGRESS_GW_IP} -X PUT -H "X-httpbin: true" http://www.example.com:80/bytes/5
       

   • HTTPS:

       curl -vik --resolve www.example.com:443:${INGRESS_GW_IP} -X PUT -H "X-httpbin: true" https://www.example.com:443/bytes/5
       

Cleanup link

You can optionally remove the resources that you set up as part of this guide.

1. Delete the external auth resources that you created.

     kubectl -n httpbin delete ExtAuthServer opa-ext-auth-server 
   kubectl -n httpbin delete ExtAuthPolicy opa-server 
     

2. Optional: If you no longer need your OPA server, delete it.

     kubectl delete all -l app=opa 
     

Update OPA config link

To update OPA config after initially deploying the OPA server sidecar, choose from the following options.

• Redeploy your OPA server installation with the new OPA config.
• Create a Kubernetes config map with the new OPA config and restart the OPA server, as show in the following steps.

Steps for creating and updating config maps:

1. Follow Steps 1 - 3 of Bundle Rego rules to create, bundle, and store your Rego rules in a cloud provider.

2. Create a Kubernetes config map that refers to your bundle.

     kubectl apply  -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: opa-config
     namespace: gloo-mesh
   data:
     config.yaml: |
       services:
         gcs:
           url: https://storage.googleapis.com/storage/v1/b/test-opa-bundles
       bundles:
         bundle:
           service: gcs
           resource: 'bundle.tar.gz?alt=media'
   EOF
     

   Review the following table to understand this configuration.

   Setting	Description
   namespace	Create the config map in the same namespace as the OPA server, such as gloo-mesh.
   config.yaml	Enter the bundle configuration details that you created as part of uploading your bundle to the cloud provider. For more information, see the OPA implementation docs.
   services	Provide the details of the cloud service where the bundle is located. In this example, the bundle is in a Google Cloud Storage (gcs) bucket at a test URL. The test URL is public, but you can also set up secure access with the credentials settings.
   bundles	Provide the details about the particular bundle that you want to use. In this example, the bundle is the bundle.tar.gz bundle that you previously created.
   Other settings	If you use other features, you can configure those settings in the config map. Common settings include signed bundles and returning decision logs. For more information, see the OPA implementation docs.

3. Update the OPA server to refer to the OPA config map.

   1. Get your current deployment configuration.

        kubectl get deployment -l app=opa -A -o yaml  > opa-deploy.yaml
      open opa-deploy.yaml
        

   2. Edit and save the deployment configuration to add the reference to the OPA config map that you previously created.

        ...
      spec:
        containers:
          volumeMounts:
            - readOnly: true
              mountPath: /policies
              name: opa-config
        volumes:
        - name: opa-config
          configMap:
            name: opa-config
        

   3. Update the deployment.

        kubectl apply -f opa-deploy.yaml 
        

   4. Optional: Later, if you change the contents of the config map, you must restart the OPA server for the OPA config changes to take effect. Optionally, you can create the config map before enabling the OPA server sidecar and refer to the config map during the Helm installation. This way, the first time the OPA server sidecar comes up, it has the OPA config already, reducing the number of times you might have to restart the service.

        kubectl rollout restart deployment/opa -n gloo-mesh 
        

4. Confirm that the OPA server loaded the bundle that you referred to in the config map. Note that the following example command pipes the output to jq for readability.

     kubectl logs -n gloo-mesh deploy/opa  | jq
     

      {
        "level": "info",
        "msg": "Bundle loaded and activated successfully. Etag updated to CMzq8eO/p4EDEAE=.",
        "name": "httpbin",
        "plugin": "bundle",
        "time": "2023-09-20T15:54:47Z"
      }