Request retries

Set up retries for requests.

Specify the number of times and duration for the gateway to try a connection to an unresponsive backend service. You might commonly use retries alongside Timeouts to ensure that your apps are available even if they are temporarily unavailable.

This feature is experimental in the upstream Kubernetes Gateway API and subject to change.

About request retries

A request retry is the number of times a request is retried if it fails. This setting can be useful to avoid your apps from failing if they are temporarily unavailable. With retries, calls are retried a certain number of times before they are considered failed. Retries can enhance your app’s availability by making sure that calls don’t fail permanently because of transient problems, such as a temporarily overloaded service or network.

Before you begin

Set up an agentgateway proxy.
Install the httpbin sample app.

Step 1: Set up request retries

Set up retries to the sample app.

Install the experimental Kubernetes Gateway API CRDs.

kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/experimental-install.yaml

Create your retry rules. You can choose to apply a retry on an HTTPRoute by using the Kubernetes Gateway API-native approach, or to add a retry to a specific HTTPRoute rule or Gateway listener by using an AgentgatewayBackend resource.

Create an HTTPRoute that routes requests along the retry.example domain to the sample app.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: retry
  namespace: httpbin
spec:
  hostnames:
  - retry.example
  parentRefs:
  - name: agentgateway-proxy
    namespace: agentgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      port: 8000
    retry:
      attempts: 3
      backoff: 1s
      codes:
      - 500
      - 503
EOF

Review the following table to understand this configuration.

Field	Description
`hostnames`	The hostnames to match the request, such as `retry.example`.
`parentRefs`	The gateway to which the request is sent. In this example, you select the `agentgateway-proxy` gateway that you set up before you began.
`rules`	The rules to apply to requests.
`matches`	The path to match the request. In this example, you match any requests to the sample app with path prefix `/`.
`path`	The path to match the request.
`backendRefs`	The backend service to which the request is sent. In this example, you select the `httpbin` service that you set up in before you begin.
`retry.attempts`	The number of times to retry the request. In this example, you retry the request 3 times.
`retry.backoff`	The duration to wait before retrying the request. In this example, you wait 1 second before retrying the request.
`retry.codes`	The HTTP status codes for which the gateway retries the request. In this example, the request is retried if the backend returns 500 or 503.

Verify that the gateway proxy is configured to retry the request.

Port-forward the gateway proxy on port 15000.

kubectl port-forward deploy/agentgateway-proxy -n agentgateway-system 15000

Find the route configuration for the cluster in the config dump. Verify that the retry policy is set as you configured it.

Example jq command:

curl -s http://localhost:15000/config_dump | jq '[.binds[].listeners | to_entries[] | .value.routes | to_entries[] | select(.value.name == "retry" and (.value.inlinePolicies[]? | has("retry"))) | .value] | .[0]'

Example output:

http://localhost:15000/config_dump

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
...
{
  "key": "httpbin/retry.0.0.http",
  "name": "retry",
  "namespace": "httpbin",
  "hostnames": [
    "retry.example"
  ],
  "matches": [
    {
      "path": {
        "pathPrefix": "/"
      }
    }
  ],
  "backends": [
    {
      "weight": 1,
      "service": {
        "name": "httpbin/httpbin.httpbin.svc.cluster.local",
        "port": 8000
      }
    }
  ],
  "inlinePolicies": [
    {
      "retry": {
        "attempts": 3,
        "backoff": "1s",
        "codes": [
          "500 Internal Server Error",
          "503 Service Unavailable"
        ]
      }
    }
  ]
}

Create an HTTPRoute that routes requests along the retry.example domain to the sample app.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: retry
  namespace: httpbin
spec:
  hostnames:
  - retry.example
  parentRefs:
  - name: agentgateway-proxy
    namespace: agentgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      namespace: httpbin
      port: 8000
    name: http
EOF

Create an EnterpriseAgentgatewayPolicy that applies a retry policy to the HTTPRoute rule.

kubectl apply -f- <<EOF
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
  name: retry
  namespace: httpbin
spec:
  targetRefs:
  - kind: HTTPRoute
    group: gateway.networking.k8s.io
    name: retry
    sectionName: http
  traffic:
    retry:
      attempts: 3
      backoff: 1s
      codes:
      - 500
      - 503
EOF

Review the following table to understand this configuration.

Field	Description
`targetRefs.sectionName`	Select the HTTPRoute rule that you want to apply the policy to.
`retry.attempts`	The number of times to retry the request. In this example, you retry the request 3 times.
`retry.backoff`	The duration to wait before retrying the request. In this example, you wait 1 second before retrying the request.
`retry.codes`	The condition that must be met for the gateway proxy to retry the request. In this example, the request is retried if a 500 or 503 HTTP response code is returned.

Verify that the gateway proxy is configured to retry the request.

Port-forward the gateway proxy on port 15000.

kubectl port-forward deploy/agentgateway-proxy -n agentgateway-system 15000

Find the route configuration for the cluster in the config dump. Verify that the retry policy is set as you configured it.

Example jq command:

curl -s http://localhost:15000/config_dump | jq '[.. | objects | select(has("policy") and .policy.traffic.retry?.attempts == 3 and .name.name? == "retry")] | .[0]'

Example output:

http://localhost:15000/config_dump

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
   "key": "traffic/agentgateway-system/retry:retry:agentgateway-system/retry",
   "name": {
     "kind": "AgentgatewayPolicy",
     "name": "retry",
     "namespace": "agentgateway-system"
   },
   "target": {
     "route": {
       "name": "retry",
       "namespace": "agentgateway-system",
       "ruleName": "agentgateway-proxy"
     }
   },
   "policy": {
     "traffic": {
       "phase": "route",
       "retry": {
         "attempts": 3,
         "backoff": "1s",
         "codes": [
           "500 Internal Server Error",
           "503 Service Unavailable"
         ]
       }
     }
   }
 }

Create an HTTPRoute that routes requests along the retry.example domain to the sample app.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: retry
  namespace: httpbin
spec:
  hostnames:
  - retry.example
  parentRefs:
  - name: agentgateway-proxy
    namespace: agentgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: httpbin
      namespace: httpbin
      port: 8000
EOF

Create an EnterpriseAgentgatewayPolicy that applies a retry policy to the agentgateway-proxy Gateway listener. You set up this Gateway in the before you begin section.

kubectl apply -f- <<EOF
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
  name: retry
  namespace: agentgateway-system
spec:
  targetRefs:
  - kind: Gateway
    group: gateway.networking.k8s.io
    name: agentgateway-proxy
    sectionName: http
  traffic:
    retry:
      attempts: 3
      backoff: 1s
      codes:
      - 500
      - 503
EOF

Field	Description
`targetRefs.sectionName`	Select the Gateway listener that you want to apply the policy to.
`retry.attempts`	The number of times to retry the request. In this example, you retry the request 3 times.
`retry.backoff`	The duration to wait before retrying the request. In this example, you wait 1 second before retrying the request.
`retry.codes`	The condition that must be met for the gateway proxy to retry the request. In this example, the request is retried if a 500 or 503 HTTP response code is returned.

Verify that the gateway proxy is configured to retry the request.

Port-forward the gateway proxy on port 15000.

kubectl port-forward deploy/agentgateway-proxy -n agentgateway-system 15000

Find the route configuration for the cluster in the config dump. Verify that the retry policy is set as you configured it.

Example jq command:

curl -s http://localhost:15000/config_dump | jq '[.. | objects | select(has("policy") and .policy.traffic.retry?.attempts == 3 and .name.name? == "retry")] | .[0]'

Example output:

http://localhost:15000/config_dump

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
   "key": "traffic/agentgateway-system/retry:retry:agentgateway-system/agentgateway-proxy/http",
   "name": {
     "kind": "AgentgatewayPolicy",
     "name": "retry",
     "namespace": "agentgateway-system"
   },
   "target": {
     "gateway": {
       "gatewayName": "agentgateway-proxy",
       "gatewayNamespace": "agentgateway-system",
       "listenerName": "http"
     }
   },
   "policy": {
     "traffic": {
       "phase": "route",
       "retry": {
         "attempts": 3,
         "backoff": "1s",
         "codes": [
           "500 Internal Server Error",
           "503 Service Unavailable"
         ]
       }
     }
   }
 }

Send a request to the sample app. Verify that the request succeeds.

curl -vi http://$INGRESS_GW_ADDRESS:80/headers -H "host: retry.example"

curl -vi localhost:8080/headers -H "host: retry.example"

Example output:

...
< HTTP/1.1 200 OK
...

Verify that the request was not retried.

kubectl logs -n agentgateway-system \
-l gateway.networking.k8s.io/gateway-name=agentgateway-proxy \
--tail=1 | grep -E 'retry.example'

Example output:

info	request gateway=agentgateway-system/agentgateway-proxy
listener=http route=httpbin/retry endpoint=10.244.0.13:8080
src.addr=127.0.0.1:34300 http.method=GET http.host=retry.example
http.path=/headers http.version=HTTP/1.1 http.status=200
protocol=http duration=0ms

Step 2: Trigger a retry

Simulate a failure for the sample app so that you can verify that the request is retried.

Send another request to the httpbin app along the /status/500 path. This path returns a 500 HTTP response code. Because the agentgateway proxy is configured to retry a request when a 500 HTTP response code is received, the proxy starts retrying the request.
curl -vi http://$INGRESS_GW_ADDRESS:80/status/500 -H "host: retry.example:80"
curl -vi localhost:8080/status/500 -H "host: retry.example"
Example output:
```
...
< HTTP/1.1 500 Internal Server Error
...
```

Verify that the request was retried. Look for retry.attempt=3 in the output.

kubectl logs -n agentgateway-system \
-l gateway.networking.k8s.io/gateway-name=agentgateway-proxy \
--tail=1 | grep -E 'retry.example'

Example output:

info	request gateway=agentgateway-system/agentgateway-proxy
listener=http route=httpbin/retry endpoint=10.244.0.21:8080
src.addr=127.0.0.1:59284 http.method=GET http.host=retry.example
http.path=/status/500 http.version=HTTP/1.1 http.status=500
protocol=http retry.attempt=3 duration=1ms

Cleanup

You can remove the resources that you created in this guide.

Delete the HTTPRoute resource.

kubectl delete httproute retry -n httpbin

If you created an EnterpriseAgentgatewayPolicy, delete it from the namespace you created it in.

kubectl delete EnterpriseAgentgatewayPolicy retry -n httpbin
kubectl delete EnterpriseAgentgatewayPolicy retry -n agentgateway-system

About Per-try timeout