TCP connection

Set up connection pool settings such as keepalive for TCP protocols.

With a keepalive connection policy, the kernel sends probe packets with only an acknowledgement flag (ACK) to the TCP socket of the destination. If the destination returns the packet with an acknowledgement flag (ACK), the connection is determined to be alive. If not, the probe can fail a certain number of times before the connection is considered dead and the destination is removed from the load balancing pool.

You can use the connection policy to keep connections alive and avoid 503 errors. Consider the setup in the following figure.

Connection policy example

In the web workspace, your web app needs to call the recommendation app, which is in a different cluster and workspace. Because you have a Gloo virtual destination, the web app can easily reach the recommendation app by calling recommendation.global. The request goes to the load balancer that backs the east-west gateway, such as an AWS ELB or NLB or the Azure load balancer. Then, the east-west gateway sends the request to the recommendation app.

Say that your web app sends a request to the recommendation app infrequently. Because traffic is not sent for a while, the load balancer hangs and terminates the connection. However, the web app client still is connected to the gateway. If the client now sends a request to the web app, the connection fails and the client gets back a 503 HTTP response.

To avoid this scenario, you apply a Gloo connection policy to your virtual destination that sends a keepalive request to the target endpoint every minute. Now, the load balancer no longer closes the connection. You can set a Gloo connection policy for every virtual destination to help prevent load balancers from closing connections.

For more information, see the following resources.

Before you begin

This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.
  1. Complete the multicluster getting started guide to set up the following testing environment.
    • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
    • The Gloo Platform CLI, meshctl, along with other CLI tools such as kubectl and istioctl.
    • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
    • Istio installed in the workload clusters.
    • A simple Gloo workspace setup.
  2. Install Bookinfo and other sample apps.

Configure connection policies

You can apply a connection policy at the destination level. For more information, see Applying policies.

The following example applies a TCP keepalive configuration for every destination in the workspace.

apiVersion: resilience.policy.gloo.solo.io/v2
kind: ConnectionPolicy
metadata:
  name: tcp-keepalive
  namespace: bookinfo
spec:
  applyToDestinations:
  - kind: VIRTUAL_DESTINATION
    selector: {}
  config:
    tcp:
      tcpKeepalive:
        time: "700s"
        probes: 20
        interval: "10s"

Review the following table to understand this configuration. For more information, see the API docs.

Setting Description
applyToDestinations Configure which destinations to apply the policy to, by using labels. Destinations can be a Kubernetes service, VirtualDestination, or ExternalService. If you do not specify any destinations or routes, the rate limit policy applies to all destinations in the workspace by default. If you do not specify any destinations but you do specify a route, the rate limit applies to the route but to no destinations.
config Configure the connection settings of the low-level networking protocol to apply to the selected destinations. Only the tcp protocol is supported.
maxConnections Set the maximum number of connections to the destination host. In this example, no timeout is set, so the default is 2^32-1.
connectTimeout Set the TCP connection timeout, which must be greater than or equal to 1ms. In this example, no timeout is set, so the default is 10s.
tcpKeepalive Set the TCP keepalive settings.
time The duration of time that a connection can idle before keepalive probes are sent. Set this value as an integer plus a unit of time, in the format 1h, 1m, 1s, or 1ms. The value must be at least 1ms, and defaults to the OS level of 7200s (or 2 hours) in Linux. This example sets the time to a shorter duration of 700s.
probes Set the maximum number of TCP keepalive probes to send before determining that connection is dead. If omitted, the value defaults to the OS configuration, which is typically 9 in Linux. This example allows more probes, 20.
interval The duration of time between keepalive probes. Set this value as an integer plus a unit of time, in the format 1h, 1m, 1s, or 1ms. The value must be at least 1ms, and defaults to the OS level of 75s in Linux. This example sets the interval to a shorter duration of 10s.

Verify connection policies

  1. Create a virtual destination for the ratings app.

    kubectl apply --context $REMOTE_CONTEXT1 -n bookinfo -f- <<EOF
    apiVersion: networking.gloo.solo.io/v2
    kind: VirtualDestination
    metadata:
      name: ratings-global
      namespace: bookinfo
    spec:
      hosts:
      - ratings.global
      ports:
      - number: 80
        protocol: HTTP
        targetPort:
          name: http
      services:
      - labels:
          app: ratings
    EOF
    
  2. Apply the example connection policy that selects all virtual destinations.

    apiVersion: resilience.policy.gloo.solo.io/v2
    kind: ConnectionPolicy
    metadata:
      name: tcp-keepalive
      namespace: bookinfo
    spec:
      applyToDestinations:
      - kind: VIRTUAL_DESTINATION
        selector: {}
      config:
        tcp:
          tcpKeepalive:
            time: "700s"
            probes: 20
            interval: "10s"
    
  3. Verify that an Istio destination rule is created for the ratings virtual destination.

    kubectl get destinationrule -n bookinfo --context $REMOTE_CONTEXT1
    

    Example output:

    NAME                                                              HOST             AGE
    ratings-global-virtual-destinat-2ab46384c8b40be3cfab9740ac8fb2c   ratings.global   11s
    
  4. Describe the Istio destination rule.

    kubectl describe destinationrule <destination-rule> -n bookinfo --context $REMOTE_CONTEXT1
    

    In the output, verify that the Connection Pool settings include the Tcp Keepalive that your policy configures.

    Traffic Policy:
     Port Level Settings:
       Connection Pool:
         Tcp:
           Tcp Keepalive:
             Interval:  10s
             Probes:    20
             Time:      700s
    
  5. Optional: Clean up the resources that you created.

    kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete virtualdestination ratings-global
    kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete connectionpolicy tcp-keepalive