Set separate timeouts for retries.

About per-try timeouts

The per-retry timeout allows you to set a timeout for retried requests. If the timeout expires, Envoy cancels the retry attempt and immediately retries on another upstream host.

By default, Envoy has a default overall request timeout of 15 seconds. A request timeout represents the time Envoy waits for the entire request to complete, including retries. Without a per-try timeout, retries might take longer than the overall request timeout, and therefore might not be executed as the request times out before the retry attempts can be performed. You can configure a larger request timeout to account for this case. However, you can also define timeouts for each retry so that you can protect against slow retry attempts from consuming the entire request timeout.

Note that if you configured a global request timeout, the per-try timeout must be less than the global request timeout.

Per-try timeouts can be configured on an HTTPRoute directly. To enable per-try timeouts on a Gateway listener level, use a GlooTrafficPolicy instead.

Before you begin

  1. Follow the Get started guide to install Gloo Gateway.

  2. Follow the Sample app guide to create a gateway proxy with an HTTP listener and deploy the httpbin sample app.

  3. Get the external address of the gateway and save it in an environment variable.

      export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS  
  
      kubectl port-forward deployment/http -n gloo-system 8080:8080

Set up per-retry timeouts

  1. Install the experimental Kubernetes Gateway API CRDs.

      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/experimental-install.yaml

  2. Configure the per-retry timeout. You can apply the timeout to an HTTPRoute, or Gateway listener.

    Use the timeouts.backendRequest field to configure the per-try timeout. Note that you must set a retry policy also to configure a per-try timeout.

      kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: retry
  namespace: httpbin
spec:
  hostnames:
  - retry.example
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: http
    namespace: gloo-system
  rules:
  - matches: 
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 8000
    retry:
      attempts: 3
      backoff: 1s  
    timeouts:
      backendRequest: 5s 
EOF
    1. Create an HTTPRoute to route requests along the retry.example domain to the httpbin app. Note that you add a name timeout to your HTTPRoute rule so that you can configure the per-try timeout for that rule later.
        kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: retry
  namespace: httpbin
spec:
  hostnames:
  - retry.example
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: http
    namespace: gloo-system
  rules:
  - matches: 
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 8000
    name: timeout 
EOF
    2. Create a GlooTrafficPolicy to configure the per-try timeout. In this example, the per-try timeout is set to 5 seconds and assigned to the timeout HTTPRoute rule. Note that you must set a retry policy also to apply a per-try timeout.
        kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooTrafficPolicy
metadata:
  name: retry
  namespace: httpbin
spec:
  targetRefs:
  - kind: HTTPRoute
    group: gateway.networking.k8s.io
    name: retry
    sectionName: timeout
  retry:
    attempts: 3
    perTryTimeout: 5s
    statusCodes:
    - 517
EOF
    1. Create an HTTPRoute to route requests along the retry.example domain to the httpbin app.
        kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: retry
  namespace: httpbin
spec:
  hostnames:
  - retry.example
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: http
    namespace: gloo-system
  rules:
  - matches: 
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 8000
EOF
    2. Create GlooTrafficPolicy to configure the per-try timeout. In this example, the per-try timeout is set to 5 seconds and assigned to the http Gateway listener that you set up as part of the before you begin section. Note that you must set a retry policy also to apply a per-try timeout.
        kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1alpha1
kind: GlooTrafficPolicy
metadata:
  name: retry
  namespace: gloo-system
spec:
  targetRefs:
  - kind: Gateway
    group: gateway.networking.k8s.io
    name: http
    sectionName: http
  retry:
    attempts: 2
    perTryTimeout: 5s
    statusCodes:
    - 517
EOF

  3. Send a request to the httpbin app along the retry.example domain. Verify that the X-Envoy-Expected-Rq-Timeout-Ms header is set to the 5 second timeout that you configured.

      curl -vi http://$INGRESS_GW_ADDRESS:8080/anything -H "host: retry.example:8080"
  
      curl -vi localhost:8080/anything -H "host: retry.example"

    Example output:

      ...
{
 "args": {},
 "headers": {
   "Accept": [
     "*/*"
   ],
   "Host": [
     "retry.example"
   ],
   "User-Agent": [
     "curl/8.7.1"
   ],
   "X-Envoy-Expected-Rq-Timeout-Ms": [
     "5000"
   ],
   "X-Envoy-External-Address": [
     "127.0.0.1"
   ],
   "X-Forwarded-For": [
     "10.244.0.55"
   ],
   "X-Forwarded-Proto": [
     "http"
   ],
   "X-Request-Id": [
     "9178dc39-297f-438a-8bd9-4e8203c06b59"
   ]
},

  4. Verify that the gateway proxy is configured with the per-try timeout.

    1. Port-forward the gateway proxy on port 19000.

        kubectl port-forward deployment/http -n gloo-system 19000

    2. Get the configuration of your gateway proxy as a config dump.

        curl -X POST 127.0.0.1:19000/config_dump\?include_eds > gateway-config.json

    3. Open the config dump and find the route configuration for the kube_default_reviews_9080 Envoy cluster on the listener~8080~retry_example virtual host. Verify that the retry policy is set as you configured it.

      Example jq command:

        jq '.configs[] | select(."@type" == "type.googleapis.com/envoy.admin.v3.RoutesConfigDump") | .dynamic_route_configs[].route_config.virtual_hosts[] | select(.routes[].route.cluster == "kube_httpbin_httpbin_8000")' gateway-config.json

      Example output:

        "routes": [
  {
    "match": {
      "prefix": "/"
    },
    "route": {
      "cluster": "kube_httpbin_httpbin_8000",
      "timeout": "5s",
      "retry_policy": {
        "retry_on": "cancelled,connect-failure,refused-stream,retriable-headers,retriable-status-codes,unavailable",
        "num_retries": 3,
        "per_try_timeout": "5s",
        "retry_back_off": {
          "base_interval": "1s"
        }
      },
      "cluster_not_found_response_code": "INTERNAL_SERVER_ERROR"
    },
    "name": "listener~8080~retry_example-route-0-httproute-retry-httpbin-0-0-matcher-0"
  }
]

Cleanup

You can remove the resources that you created in this guide.
  kubectl delete GlooTrafficPolicy retry -n httpbin
kubectl delete GlooTrafficPolicy retry -n gloo-system
kubectl delete httproute retry -n httpbin