On this page

Header manipulation

Set up an external processing (extProc) server that manipulates request headers for a sample app.

info

This feature is an Enterprise-only feature that requires a Gloo Gateway Enterprise license.

Before you begin

Follow the Get started guide to install Gloo Gateway, set up a gateway resource, and deploy the httpbin sample app.

Get the external address of the gateway and save it in an environment variable.

  export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system gloo-proxy-http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

  kubectl port-forward deployment/gloo-proxy-http -n gloo-system 8080:8080

Enable extProc in Gloo Gateway

You can enable external processing for all routes that the gateway serves by using the Gloo Gateway Settings resource.

Edit the Gloo Gateway Settings resource and add your extProc settings.
```
  kubectl edit settings default -n gloo-system
  
```

Add the following extProc settings to the spec section. This example configures the standard extProc filter, which runs in the middle of the Envoy filter chain. Gloo Gateway also supports extProcEarly (runs early in the filter chain) and extProcLate (runs as the final filter before a request leaves Envoy). For more information, see ExtProc filter variants.

  
spec:
  extProc:
    allowModeOverride: false
    failureModeAllow: false
    filterStage:
      predicate: After
      stage: AuthZStage
    grpcService:
      extProcServerRef:
        name: ext-proc-grpc
        namespace: gloo-system
    processingMode:
      requestHeaderMode: SEND
      responseHeaderMode: SKIP

Setting	Description
`allowModeOverride`	Allow the extProc server to override the processing mode settings that you set. Default value is `false`.
`failureModeAllow`	Allow the extProc server to continue when an error is detected during external processing. If set to `true`, the extProc server continues. If set to `false`, external processing is stopped and an error is returned to the Envoy proxy.
`filterStage.predicate`	How to apply the filter relative to `filterStage.stage`. Applies to `extProcEarly` and `extProc`. Has no effect on `extProcLate`, which always runs as the final filter.
`filterStage.stage`	The stage in the filter chain where you want to enable external processing. In this example, external processing is added after the authorization stage.
`grpcService.extProcServerRef`	The name and namespace of the Upstream resource that represents your external processing server.
`processingMode.requestHeaderMode`	Send (`SEND`) or skip sending (`SKIP`) request header information to the extProc server.
`processingMode.responseHeaderMode`	Send (`SEND`) or skip sending (`SKIP`) response header information to the extProc server.

Deploy an extProc server

Set up the extProc server. This example uses a prebuilt extProc server that manipulates request and response headers based on instructions that are sent in an instructions header.

  kubectl apply -n gloo-system -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ext-proc-grpc
spec:
  selector:
    matchLabels:
      app: ext-proc-grpc
  replicas: 1
  template:
    metadata:
      labels:
        app: ext-proc-grpc
    spec:
      containers:
        - name: ext-proc-grpc
          image: gcr.io/solo-test-236622/ext-proc-example-basic-sink:0.0.5
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 18080
---
apiVersion: v1
kind: Service
metadata:
  name: ext-proc-grpc
  labels:
    app: ext-proc-grpc
  annotations:
    gloo.solo.io/h2_service: "true"
spec:
  ports:
  - port: 4444
    targetPort: 18080
    protocol: TCP
  selector:
    app: ext-proc-grpc
EOF

The instructions header must be provided as a JSON string in the following format:

  {
  "addHeaders": {
    "header1": "value1",
    "header2": "value2"
  },
  "removeHeaders": [ "header3", "header4" ],
  }
}

Verify that the extProc server is up and running.

  kubectl get pods -n gloo-system | grep ext-proc-grpc

Create an Upstream resource that represents the extProc server. Note that you must include the useHttp2: true setting, because the extProc server requires the HTTP/2 protocol.

info

If you set gloo.discovery.enabled=true during the Gloo Gateway installation, the Upstream is automatically created for you.

  kubectl apply -n gloo-system -f- <<EOF        
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  labels:
    app: ext-proc-grpc
    discovered_by: kubernetesplugin
  name: ext-proc-grpc
  namespace: gloo-system
spec:
  discoveryMetadata: {}
  useHttp2: true
  kube:
    selector:
      app: ext-proc-grpc
    serviceName: ext-proc-grpc
    serviceNamespace: gloo-system
    servicePort: 4444
EOF

Verify extProc

Send a simple request to the httpbin app and make sure that you get back a 200 HTTP response code. The following request passes in two headers header1 and header2 that you see in your response.

  curl -vik http://$INGRESS_GW_ADDRESS:8080/headers -H "host: www.example.com:8080" -H "header1: value1" -H "header2: value2"

  curl -vik localhost:8080/headers -H "host: www.example.com:8080" -H "header1: value1" -H "header2: value2"

Example output:

   {
     "headers": {
       "Accept": [
         "*/*"
       ],
       "Header1": [
         "value1"
       ],
       "Header2": [
         "value2"
       ],
       "Host": [
         "www.example.com:8080"
       ],
       "User-Agent": [
         "curl/7.77.0"
       ],
       "X-Envoy-Expected-Rq-Timeout-Ms": [
         "15000"
       ],
       "X-Forwarded-Proto": [
         "http"
       ],
       "X-Request-Id": [
         "5ebe6cb0-24fc-427f-9bb7-8aa47e360c94"
      ]
     }
   }

Send another request to the httpbin app. This time, you pass along instructions for the extProc server in an instruction header. In the following example, you use the extProc server to add the header3 header, and to remove the header2 header.

  curl -vik http://$INGRESS_GW_ADDRESS:8080/headers -H "host: www.example.com:8080" -H "header1: value1" -H "header2: value2" -H 'instructions: {"addHeaders":{"header3":"value3","header4":"value4"},"removeHeaders":["instructions", "header2"]}'

  curl -vik localhost:8080/headers -H "host: www.example.com:8080" -H "header1: value1" -H "header2: value2" -H 'instructions: {"addHeaders":{"header3":"value3","header4":"value4"},"removeHeaders":["instructions", "header2"]}'

Example output:

   {
     "headers": {
       "Accept": [
         "*/*"
       ],
       "Header1": [
         "value1"
       ],
       "Header3": [
         "value3"
       ],
       "Header4": [
         "value4"
       ],
       "Host": [
         "www.example.com:8080"
       ],
       "User-Agent": [
         "curl/7.77.0"
       ],
       "X-Envoy-Expected-Rq-Timeout-Ms": [
         "15000"
       ],
       "X-Forwarded-Proto": [
         "http"
       ],
       "X-Request-Id": [
         "a3fd7124-8485-45d5-8326-cf025bc7a01c"
       ]
     }
   }

Overwrite gateway settings with RouteOptions

Create a RouteOption resource that overwrites the gateway-level settings. In this example, you want to skip sending request headers to the extProc server.

  kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: extproc
  namespace: httpbin
spec:
  options:
    extProc:
      overrides: 
        processingMode: 
          requestHeaderMode: SKIP
EOF

Create an HTTPRoute resource for the httpbin app that references the RouteOption resource that you created for the extproc.example domain.

  kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-extproc
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: gloo-system
  hostnames:
    - extproc.example
  rules:
    - filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.solo.io
            kind: RouteOption
            name: extproc
      backendRefs:
        - name: httpbin
          port: 8000
EOF

Send a request to the httpbin app and pass along instructions for the extProc server in an instruction header. Verify that the request header information is not processed because you configured Gloo Gateway to skip sending request headers to the extProc server for that route. For example, the header2 header is not removed, and header3 and header4 are not added. Instead, you get back the instructions in an instructions header.

  curl -vik http://$INGRESS_GW_ADDRESS:8080/headers -H "host: extproc.example:8080" -H "header1: value1" -H "header2: value2" -H 'instructions: {"addHeaders":{"header3":"value3","header4":"value4"},"removeHeaders":["instructions", "header2"]}'

  curl -vik localhost:8080/headers -H "host: extproc.example:8080" -H "header1: value1" -H "header2: value2" -H 'instructions: {"addHeaders":{"header3":"value3","header4":"value4"},"removeHeaders":["instructions", "header2"]}'

Example output:

   {
     "headers": {
       "Accept": [
         "*/*"
       ],
       "Header1": [
         "value1"
       ],
       "Header2": [
         "value2"
       ],
       "Host": [
         "extproc.example:8080"
       ],
       "Instructions": [
         "{\"addHeaders\":{\"header3\":\"value3\",\"header4\":\"value4\"},\"removeHeaders\":[\"instructions\", \"header2\"]}"
       ],
       "User-Agent": [
         "curl/7.77.0"
       ],
       "X-Envoy-Expected-Rq-Timeout-Ms": [
         "15000"
       ],
       "X-Forwarded-Proto": [
         "http"
       ],
       "X-Request-Id": [
         "c896a2ec-2dc2-4f91-8836-5377f2c53e7d"
       ]
     }
   }

Change the RouteOption resource and set processingMode.requestHeaderMode: SEND to send request headers for that route.

  kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: extproc
  namespace: httpbin
spec:
  options:
    extProc:
      overrides: 
        processingMode: 
          requestHeaderMode: SEND
EOF

Send another request to the httpbin app. Verify that your instructions are now processed, and header3 header is added and the header2 header is removed.

  curl -vik http://$INGRESS_GW_ADDRESS:8080/headers -H "host: extproc.example:8080" -H "header1: value1" -H "header2: value2" -H 'instructions: {"addHeaders":{"header3":"value3","header4":"value4"},"removeHeaders":["instructions", "header2"]}'

  curl -vik localhost:8080/headers -H "host: extproc.example:8080" -H "header1: value1" -H "header2: value2" -H 'instructions: {"addHeaders":{"header3":"value3","header4":"value4"},"removeHeaders":["instructions", "header2"]}'

Example output:

   {
     "headers": {
       "Accept": [
         "*/*"
       ],
       "Header1": [
         "value1"
       ],
       "Header3": [
         "value3"
       ],
       "Header4": [
         "value4"
       ],
       "Host": [
         "extproc.example:8080"
       ],
       "User-Agent": [
         "curl/7.77.0"
       ],
       "X-Envoy-Expected-Rq-Timeout-Ms": [
         "15000"
       ],
       "X-Forwarded-Proto": [
        "http"
       ],
       "X-Request-Id": [
         "943e4d99-f6c7-465e-9a25-be760220f7ec"
       ]
     }
   }

Use multiple extProc filter variants

You can configure extProcEarly and extProcLate alongside extProc to run multiple external processors at different stages of the filter chain. For example, you might want to debug your extProc server by logging requests at both the earliest and latest stages so that you can compare what changed in between. You can also use this setup to integrate with different extProc servers.

Update the default Settings resource to configure all three extProc variants. In this example, all extProc variants use the same extProc server. However, you can configure a specific extProc server for each phase. The extProc stages are processed as follows:

extProcEarly: Requests are modified before the Fault Envoy filter, which is the first filter in the filter chain.
extProc: Requests are modified after the AuthZ Envoy filter.
extProcLate: Requests are modified in the upstream_http_filter that is part of the Router phase. Note that although you must provide a filterStage setting, this setting is ignored as the extProcLate variant is always executed as part of the upstream_http_filter filter.

  kubectl edit settings default -n gloo-system

Update your settings as follows:

  extProcEarly:
  grpcService:
    extProcServerRef:
      name: ext-proc-grpc
      namespace: gloo-system
  filterStage:
    stage: FaultStage
    predicate: Before
  processingMode:
    requestHeaderMode: SEND
    responseHeaderMode: SEND
extProc:
  grpcService:
    extProcServerRef:
      name: ext-proc-grpc
      namespace: gloo-system
  filterStage:
    stage: AuthZStage
    predicate: After
  failureModeAllow: false
  allowModeOverride: false
  processingMode:
    requestHeaderMode: SEND
    responseHeaderMode: SKIP
extProcLate:
  # The filter stage is ignored as it is always executed in the upstream_http_filter.
  filterStage:
    stage: AuthZStage
    predicate: After
  grpcService:
    extProcServerRef:
      name: ext-proc-grpc
      namespace: gloo-system
  processingMode:
    requestHeaderMode: SEND
    responseHeaderMode: SEND

Create an HTTPRoute resource to expose the httpbin app on the gateway.

  kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-extproc
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: gloo-system
  hostnames:
    - extproc.example
  rules:
    - backendRefs:
        - name: httpbin
          port: 8000
EOF

Send a request to the httpbin app.

  curl -vik http://$INGRESS_GW_ADDRESS:8080/headers -H "host: extproc.example:8080" -H "header1: value1"

  curl -vik localhost:8080/headers -H "host: extproc.example:8080" -H "header1: value1"

Verify that you get back a 200 HTTP response code.

  HTTP/1.1 200 OK

Check the extProc server logs to verify that it received 3 processing requests (one for each stage). Because each variant sends request headers (requestHeaderMode: SEND), the server receives three separate gRPC processing calls per request.

  kubectl logs -l app=ext-proc-grpc -n gloo-system --tail=50

Example output:

"Process"
"Got RequestHeaders"
"Sending ProcessingResponse"
"Process"
"Got RequestHeaders"
"Sending ProcessingResponse"
"Process"
"Got RequestHeaders"
"Sending ProcessingResponse"
"Got ResponseHeaders"
"Sending ProcessingResponse"

Cleanup

Remove the resources that you created in this guide.

  kubectl delete httproute httpbin-extproc -n httpbin
kubectl delete routeoption extproc -n httpbin
kubectl delete upstream ext-proc-grpc -n gloo-system
kubectl delete deployment ext-proc-grpc -n gloo-system
kubectl delete service ext-proc-grpc -n gloo-system

Edit the Settings resource and remove the extProc settings.
```
  kubectl edit settings default -n gloo-system
  
```

Solo Enterprise for kgateway

Gloo Gateway (K8s GW API)

Gloo Edge

Header manipulation

Before you begin link

Enable extProc in Gloo Gateway link

Deploy an extProc server link

Verify extProc link

Overwrite gateway settings with RouteOptions link

Use multiple extProc filter variants link

Cleanup link

link

Before you begin

Enable extProc in Gloo Gateway

Deploy an extProc server

Verify extProc

Overwrite gateway settings with RouteOptions

Use multiple extProc filter variants

Cleanup