About

With a keepalive connection policy, the kernel sends probe packets with only an acknowledgement flag (ACK) to the TCP socket of the destination. If the destination returns the packet with an acknowledgement flag (ACK), the connection is determined to be alive. If not, the probe can fail a certain number of times before the connection is considered dead and the destination is removed from the load balancing pool.

You can use the connection policy to keep connections alive and avoid 503 errors. Consider the setup in the following figure.

Figure: Connection policy example
Figure: Connection policy example
Figure: Connection policy example
Figure: Connection policy example

In the web workspace, your web app needs to call the recommendation app, which is in a different cluster and workspace. Because you have a Gloo virtual destination, the web app can easily reach the recommendation app by calling recommendation.global. The request goes to the load balancer that backs the east-west gateway, such as an AWS ELB or NLB or the Azure load balancer. Then, the east-west gateway sends the request to the recommendation app.

Say that your web app sends a request to the recommendation app infrequently. Because traffic is not sent for a while, the load balancer hangs and terminates the connection. However, the web app client still is connected to the gateway. If the client now sends a request to the web app, the connection fails and the client gets back a 503 HTTP response.

To avoid this scenario, you apply a Gloo connection policy to your virtual destination that sends a keepalive request to the target endpoint every minute. Now, the load balancer no longer closes the connection. You can set a Gloo connection policy for every virtual destination to help prevent load balancers from closing connections.

For more information, see the following resources.

Before you begin

  1. Complete the multicluster getting started guide to set up the following testing environment.

    • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
    • The Gloo meshctl CLI, along with other CLI tools such as kubectl and istioctl.
    • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
    • Istio installed in the workload clusters.
    • A simple Gloo workspace setup.
  2. Install Bookinfo and other sample apps.

Configure connection policies

You can apply a connection policy at the destination level. For more information, see Applying policies.

The following example applies a TCP keepalive configuration for every destination in the workspace.

  apiVersion: resilience.policy.gloo.solo.io/v2
kind: ConnectionPolicy
metadata:
  name: tcp-keepalive
  namespace: bookinfo
spec:
  applyToDestinations:
  - kind: VIRTUAL_DESTINATION
    selector: {}
  config:
    tcp:
      tcpKeepalive:
        time: "700s"
        probes: 20
        interval: "10s"

Review the following table to understand this configuration. For more information, see the API docs.

SettingDescription
applyToDestinationsUse labels to apply the policy to destinations. Destinations might be a Kubernetes service, VirtualDestination, or ExternalService (if supported by the policy). If you do not specify any destinations or routes, the policy applies to all destinations in the workspace by default. If you do not specify any destinations but you do specify a route, the policy applies to the route but to no destinations.
configConfigure the connection settings of the low-level networking protocol to apply to the selected destinations. To set connection pool settings for TCP destinations, use tcp as the protocol. For HTTP connection pool settings, use http. The connection policy in this guide shows how to configure connection pool settings for a TCP destination. To find an example for an HTTP connection policy, see Connection pool settings for HTTP.
maxConnectionsSet the maximum number of connections to the destination host. In this example, no timeout is set, so the default is 2^32-1.
connectTimeoutSet the TCP connection timeout, which must be greater than or equal to 1ms. In this example, no timeout is set, so the default is 10s.
tcpKeepaliveSet the TCP keepalive settings.
timeThe duration of time that a connection can idle before keepalive probes are sent. Set this value as an integer plus a unit of time, in the format 1h, 1m, 1s, or 1ms. The value must be at least 1ms, and defaults to the OS level of 7200s (or 2 hours) in Linux. This example sets the time to a shorter duration of 700s.
probesSet the maximum number of TCP keepalive probes to send before determining that connection is dead. If omitted, the value defaults to the OS configuration, which is typically 9 in Linux. This example allows more probes, 20.
intervalThe duration of time between keepalive probes. Set this value as an integer plus a unit of time, in the format 1h, 1m, 1s, or 1ms. The value must be at least 1ms, and defaults to the OS level of 75s in Linux. This example sets the interval to a shorter duration of 10s.

Verify connection policies

  1. Create a virtual destination for the ratings app.

      kubectl apply --context $REMOTE_CONTEXT1 -n bookinfo -f- <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
  name: ratings-global
  namespace: bookinfo
spec:
  hosts:
  - ratings.global
  ports:
  - number: 80
    protocol: HTTP
    targetPort:
      name: http
  services:
  - labels:
      app: ratings
EOF

  2. Apply the example connection policy that selects all virtual destinations.

      apiVersion: resilience.policy.gloo.solo.io/v2
kind: ConnectionPolicy
metadata:
  name: tcp-keepalive
  namespace: bookinfo
spec:
  applyToDestinations:
  - kind: VIRTUAL_DESTINATION
    selector: {}
  config:
    tcp:
      tcpKeepalive:
        time: "700s"
        probes: 20
        interval: "10s"

  3. Verify that an Istio destination rule is created for the ratings virtual destination.

      kubectl get destinationrule -n bookinfo --context $REMOTE_CONTEXT1

    Example output:

      NAME                                                              HOST             AGE
ratings-global-virtual-destinat-2ab46384c8b40be3cfab9740ac8fb2c   ratings.global   11s

  4. Describe the Istio destination rule.

      kubectl describe destinationrule <destination-rule> -n bookinfo --context $REMOTE_CONTEXT1

    In the output, verify that the Connection Pool settings include the Tcp Keepalive that your policy configures.

      Traffic Policy:
 Port Level Settings:
   Connection Pool:
     Tcp:
       Tcp Keepalive:
         Interval:  10s
         Probes:    20
         Time:      700s

Cleanup

You can optionally remove the resources that you set up as part of this guide.
  kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete VirtualDestination ratings-global
kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete ConnectionPolicy tcp-keepalive