Observe WAF traffic

Configure access logging, audit logging, and metrics to monitor WAF activity and understand why the WAF blocked or allowed a request.

Access logs

You can include WAF-specific fields in Envoy access logs to identify which WAF rule triggered the blocking of a request. The %RESPONSE_CODE_DETAILS% field contains the WAF rule ID that caused a block.

The following example configuration enables access logs on the gateway proxy and includes the response_code_details field in the access logs.

kubectl apply -f- <<EOF
apiVersion: gateway.kgateway.dev/v1alpha1
kind: ListenerPolicy
metadata:
  name: waf-access-logs
  namespace: kgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: http
  default:
    httpSettings:
      accessLog:
      - fileSink:
          path: /dev/stdout
          jsonFormat:
            request_id: "%REQ(X-REQUEST-ID)%"
            method: "%REQ(X-ENVOY-ORIGINAL-METHOD?:METHOD)%"
            path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
            response_code: "%RESPONSE_CODE%"
            response_code_details: "%RESPONSE_CODE_DETAILS%"
            user_agent: "%REQ(USER-AGENT)%"
            route_name: "%ROUTE_NAME%"
EOF

After you apply the policy, send a request to your WAF-protected service and review the access logs that the gateway proxy emits.

Audit logging

You can enable audit logging in the WAF engine to get more details about why the WAF engine made a certain decision. Audit logs are captured after all Coraza phases were processed and a response is returned to the caller.

Enable audit logging on the WafPolicy. The following configuration uses the minimal audit logging part (AHZ) that returns the audit headers and WAF decision only. You can also enable other parts to get more information. However, keep in mind that larger parts might expose sensitive request data.

Parts	What it adds
`AHZ`	Returns audit header and WAF decision only.
`AKHZ`	Adds matched rule details.
`ABKHZ`	Adds request headers, which can be useful for header-based rule debugging.
`C`	Adds the request body. Can contain secrets, tokens, or personal data.
`E`	Adds the response body. Can contain secrets, tokens, or personal data.
`I`	Adds multipart request body, including files and form fields. Can contain uploaded file contents.
`J`	Adds uploaded file information. Can contain uploaded file contents.

Make sure to remove audit logging when you are done troubleshooting your WAF settings.

kubectl apply -f- <<EOF
apiVersion: waf.solo.io/v1alpha1
kind: WAFPolicy
metadata:
  name: waf-policy
  namespace: kgateway-system
spec:
  ruleEngineSettings:
    inline: |
      SecRuleEngine On
      SecAuditEngine RelevantOnly
      SecAuditLog /dev/stdout
      SecAuditLogFormat JSON
      SecAuditLogParts AHZ
EOF

Send a request to your WAF-protected backend. Then, get the waf-server logs.
```
kubectl logs deploy/waf-server-enterprise-kgateway -n kgateway-system 
```

WAF server metrics

The WAF server exposes a Prometheus-compatible /metrics endpoint on port 9091. The endpoint is enabled by default.

Port-forward the WAF server metrics port.

kubectl -n kgateway-system port-forward deployment/waf-server-enterprise-kgateway 9091

Scrape the metrics endpoint.
```
curl -s http://127.0.0.1:9091/metrics
```

The following WAF-specific metrics are available:

Metric	Type	Labels	Description
`waf_server_requests_total`	Counter	`action="allow\|deny"`, `reason`	Total requests processed by the WAF server, labeled by outcome. The `reason` label can have the following values: empty for allowed requests `waf_blocked` for rule-triggered denials `invalid_policy` when a request is denied because of an invalid WAFPolicy `invalid_metadata` when invalid stream data was detected `stream_recv_error` for stream receive errors `stream_send_error` for stream send errors `unknown_request_type` for unsupported extProc request messages `internal_error` for errors that do not fall into any of the other categories
`waf_server_processing_duration_seconds`	Histogram	`action="allow\|deny"`	WAF processing time per request, broken down by outcome.
`waf_server_policy_status`	Gauge	`namespace`, `name`, `status="active\|invalid"`	Per-policy active/invalid count. Use `sum(waf_server_policy_status{status="invalid"}) > 0` to determine the number of WAFPolicy resources that are in an invalid state. If you find invalid resources, use `waf_server_policy_status{status="invalid"} == 1` to find the name and namespace of the invalid WAFPolicy. Then, review the WAFPolicy to correct the invalid configuration.

The server also exposes standard Go runtime (go_*) and process-level (process_*) metrics.

To disable the metrics endpoint, set metrics.enabled: false in your EnterpriseKgatewayParameters.

kubectl apply -f- <<EOF
apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayParameters
metadata:
  name: enterprise-kgateway-params
  namespace: kgateway-system
spec:
  kube:
    sharedExtensions:
      waf:
        enabled: true
        metrics:
          enabled: false
EOF

Gateway proxy metrics

Use Envoy proxy metrics to investigate request timeouts, ext-proc connectivity issues, and WAF execution problems.

Port-forward the gateway proxy admin port.

kubectl -n kgateway-system port-forward deployment/http 19000

Search the metrics output for ext-proc and WAF-related signals.

curl -s http://127.0.0.1:19000/stats | grep 'ext_proc'
curl -s http://127.0.0.1:19000/stats | grep 'waf-server'

Controller metrics

Use these steps to identify issues with the Solo Enterprise for kgateway controller, including translations and xDS errors.

Port-forward the Solo Enterprise for kgateway controller on port 9092.

kubectl -n kgateway-system port-forward deployment/enterprise-kgateway 9092

Open the metrics endpoint and look for controller metrics, such as enterprise_kgateway_controller_, enterprise_kgateway_translator_, enterprise_kgateway_xds_, and solo-waf metrics.

You can also curl the metrics endpoint as follows.

curl -s http://127.0.0.1:9092/metrics | grep 'enterprise_kgateway_controller_'
curl -s http://127.0.0.1:9092/metrics | grep 'enterprise_kgateway_translator_'
curl -s http://127.0.0.1:9092/metrics | grep 'enterprise_kgateway_xds_'
curl -s http://127.0.0.1:9092/metrics | grep 'controller="solo-waf"'

Debug WAF