AdaptiveRequestConcurrencyPolicy

Proto: adaptive_request_concurrency_policy.proto

Package: resilience.policy.gloo.solo.io

Dynamically adjust the maximum number of concurrent requests that can be sent to a destination.

By setting up adaptive concurrency, you can help protect your APIs from being overwhelmed by too many concurrent requests. In the policy, you set up the parameters to control how many concurrent requests are allowed. Then, the policy applies the adaptive concurrency filter, which automatically adjusts the rate of requests sent to a backend service based on the observed latency of the responses from that service. It calculates the concurrency values by sampling the latency of completed requests, and comparing the measured samples in a time window against the expected latency for hosts in the cluster.

The filter uses a gradient controller to calculate forwarding decisions based on an ideal, minimum round-trip time (minRTT) for a service. The controller periodically measures the minRTT by allowing only a very low outstanding request count to a service, and measuring the latency under these ideal conditions. The controller then uses the minRTT value to calculate the concurrency limit. For more information about the formulas that the controller calculates, see the Envoy adaptive concurrency docs.

Adaptive request concurrency policies are applied at the Destination level. The filter is applied to the Envoy listener and contains only listener-level configuration, and no route-level configuration.

Example link

This example applies to the httpbin destination and dynamically adjusts its concurrency limits based on network conditions. The settings in the policy configure the following:

• Cap the concurrency limit to a maximum of 6 connections (maxConcurrencyLimit: 6), in the case that the calculated concurrency limit exceeds this value.
• When the service has 6 concurrent connections, return a 503 HTTP response code to any new client requests (concurrencyLimitExceededStatus: 503).
• When measuring the minRTT to determine the concurrency limit:
  • Gather 1 request latency sample (requestCount: 1) in a timeframe of 3000ms (concurrencyUpdateIntervalMillis: 3000).
  • Recalculate the minRTT every 1000ms (intervalMillis: 1000).
  • Until the latest minRTT measurement is complete, temporarily set the concurrency limit to 6 connections (minConcurrency: 6).
  • Do not add a jitter (random delay) to the start of each minRTT measurement (jitterPercentile: 0).
  • Do not add a buffer (to stabilize natural variability in latency) to the measured minRTT (bufferPercentile: 0).

  apiVersion: resilience.policy.gloo.solo.io/v2
kind: AdaptiveRequestConcurrencyPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: simple-adaptive-request-concurrency-policy
  namespace: httpbin
spec:
  applyToDestinations:
  - port:
      number: 8000
    selector:
      labels:
        app: httpbin
  config:
    concurrencyLimitExceededStatus: 503
    concurrencyUpdateIntervalMillis: 3000
    maxConcurrencyLimit: 6
    minRttCalcParams:
      bufferPercentile: 0
      intervalMillis: 1000
      jitterPercentile: 0
      minConcurrency: 6
      requestCount: 1
  

AdaptiveRequestConcurrencyPolicyReport link

AdaptiveRequestConcurrencyPolicyReport.WorkspacesEntry link

AdaptiveRequestConcurrencyPolicySpec link

Specifications for the policy.

AdaptiveRequestConcurrencyPolicySpec.Config link

Configure how the gradient controller calculates the adaptive concurrency limit.

AdaptiveRequestConcurrencyPolicySpec.Config.MinRoundtripTimeCalculationParams link

Configure how the gradient controller calculates the minimum round-trip time (minRTT) for the destination. For more information about the minRTT formula and the following fields, see the Envoy adaptive concurrency docs. This field is required.

AdaptiveRequestConcurrencyPolicyStatus link

The status of the policy after it is applied to your Gloo environment.