Failover

Failover is an important part of building resilient apps in multicluster environments. You set up locality-aware failover by specifying regions, zones, and subzones to reroute traffic. In the event of a failure in the closest locality, responses can be served from the next closest locality.

You can use failover policies in combination with other policies, such as outlier detection or retry policies. In case of a failure, the failover policy tells Gloo Mesh which healthy destinations to reroute traffic to, based on the closest locality. The outlier detection policy tells Gloo Mesh when and for how long to remove unhealthy services. The retry policy tells Gloo Mesh how many times to retry requests before the outlier detection policy considers the request as failing and removes the service from the pool of healthy destinations.

For more information, see the following resources.

• Gloo Mesh failover API docs
• Istio docs for locality failover

Before you begin

1. Complete the multicluster getting started guide to set up the following testing environment.
   • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
   • The Gloo Platform CLI, meshctl, along with other CLI tools such as kubectl and istioctl.
   • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
   • Istio installed in the workload clusters.
   • A simple Gloo workspace setup.
2. Install Bookinfo and other sample apps.

Configure failover policies

You can apply a failover policy at the destination level. For more information, see Applying policies. Note that for one destination, you cannot apply both a failover policy that specifies zones and subzones and a failover policy that only specifies regions. For one destination, you can specify multiple failover policies that specify zones and subzones, or multiple that specify regions. However, ensure that the configuration does not overlap between multiple policies. For example, if one failover policy reroutes traffic from from us-east-1 to us-east-2, and another reroutes traffic from us-east-2 to eu-west-1, the configurations do not overlap. But if one failover policy reroutes traffic from from us-east-1 to us-east-2, and another reroutes traffic from us-east-1 to eu-west-1, then the configurations overlap, and traffic might not be correctly rerouted.

Review the following sample configuration file.

apiVersion: resilience.policy.gloo.solo.io/v2
kind: FailoverPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: locality-based-failover
  namespace: bookinfo
spec:
  applyToDestinations:
  - kind: VIRTUAL_DESTINATION
    selector: {}
  config:
    localityMappings:
    - from:
        region: us-east
      to:
      - region: us-west

Review the following table to understand this configuration. For more information, see the API docs.

Verify failover policies

You can test how failover works by opening the Bookinfo app in your browser and observing the reviews app behavior after applying various resources.

1. Verify that your clusters have topology.kubernetes.io/region locality labels. If not, see Configure the locality labels for nodes.

   kubectl get nodes --context $REMOTE_CONTEXT1 -o jsonpath='{.items[*].metadata.labels}'
   kubectl get nodes --context $REMOTE_CONTEXT2 -o jsonpath='{.items[*].metadata.labels}'
   

2. Create a virtual destination for the reviews app. The virtual destination enables multicluster traffic routing.

   kubectl --context ${REMOTE_CONTEXT1} apply -f - <<EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: VirtualDestination
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: reviews-global
     namespace: bookinfo
   spec:
     hosts:
     - reviews.vd
     ports:
     - number: 80
       protocol: HTTP
       targetPort:
         name: http
     services:
     - labels:
         app: reviews
   EOF
   

3. Create an outlier detection policy to use with the failover policy so that unhealthy destinations are removed. The outlier detection policy also ensures that requests are always routed to the closest locality.

   kubectl --context ${REMOTE_CONTEXT1} apply -f - <<EOF
   apiVersion: resilience.policy.gloo.solo.io/v2
   kind: OutlierDetectionPolicy
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: outlier-detection
     namespace: bookinfo
   spec:
     applyToDestinations:
     - kind: VIRTUAL_DESTINATION
       selector: {}
     config:
       baseEjectionTime: 30s
       consecutiveErrors: 2
       interval: 1s
       maxEjectionPercent: 100
   EOF
   

4. Create the failover policy that you previously reviewed.
   If your clusters have different region labels than us-east and us-west, update those values accordingly.

   kubectl --context ${REMOTE_CONTEXT1} apply -f - <<EOF
   apiVersion: resilience.policy.gloo.solo.io/v2
   kind: FailoverPolicy
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: locality-based-failover
     namespace: bookinfo
   spec:
     applyToDestinations:
     - kind: VIRTUAL_DESTINATION
       selector: {}
     config:
       localityMappings:
       - from:
           region: us-east
         to:
         - region: us-west
   EOF
   

5. Send a request to the reviews app from the ratings app several times. Notice that although the virtual destination serves all 3 reviews versions, you only get responses with no stars (v1) and black stars (v2) from the cluster-1 cluster because the outlier detection forces all requests to be routed to the closest locality.

   kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[].metadata.name}' --context ${REMOTE_CONTEXT1}) -n bookinfo -c ratings --context ${REMOTE_CONTEXT1} -- curl -sS reviews.global:80/reviews/1 -v
   

6. Send the reviews v1 and v2 apps in cluster-1 to sleep, to mimic an app failure in a locality.

   kubectl --context ${REMOTE_CONTEXT1} -n bookinfo patch deploy reviews-v1 --patch '{"spec":{"template":{"spec":{"containers":[{"name":"reviews","command":["sleep","20h"]}]}}}}'
   kubectl --context ${REMOTE_CONTEXT1} -n bookinfo patch deploy reviews-v2 --patch '{"spec":{"template":{"spec":{"containers":[{"name":"reviews","command":["sleep","20h"]}]}}}}'
   

7. Repeat the request to the reviews app. Notice that you get responses with only red stars (v3). The unhealthy reviews v1 and v2 apps are removed, and the traffic fails over to v3 in the locality that the failover policy specifies.

   kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[].metadata.name}' --context ${REMOTE_CONTEXT1}) -n bookinfo -c ratings --context ${REMOTE_CONTEXT1} -- curl -sS reviews.global:80/reviews/1 -v
   

8. Optional: Remove the sleep command from the reviews apps to restore normal behavior.

   kubectl --context ${REMOTE_CONTEXT1} -n bookinfo patch deploy reviews-v1 --patch '{"spec":{"template":{"spec":{"containers":[{"name":"reviews","command":[]}]}}}}'
   kubectl --context ${REMOTE_CONTEXT1} -n bookinfo patch deploy reviews-v2 --patch '{"spec":{"template":{"spec":{"containers":[{"name":"reviews","command":[]}]}}}}'
   

9. Optional: Clean up the Gloo resources that you created.

   kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete VirtualDestination reviews-global
   kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete OutlierDetectionPolicy outlier-detection
   kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete FailoverPolicy locality-based-failover