Outlier detection

About link

Outlier detection is an important part of building resilient apps. An outlier detection policy sets up several conditions, such as retries and ejection percentages, that Gloo Mesh Enterprise uses to determine if a service is unhealthy. In case an unhealthy service is detected, the outlier detection policy defines how Gloo Mesh Enterprise removes (ejects) services from the pool of healthy destinations to send traffic to. Your apps then have time to recover before they are added back to the load-balancing pool and checked again for consecutive errors.

You can use failover, outlier detection, and retry timeout policies together to build a more resilient application network. For example, an outlier detection policy can remove unhealthy destinations, a failover policy can redirect traffic to healthy destinations, and a retry policy can retry requests in case of failure. Review the following table to understand what each policy does.

For more information, see the following resources.

• Gloo Mesh Enterprise outlier detection API docs
• Istio docs for circuit breaking
• Envoy docs for outlier detection

Before you begin link

1. Complete the multicluster getting started guide to set up the following testing environment.

   • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
   • The Gloo meshctl CLI, along with other CLI tools such as kubectl and istioctl.
   • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
   • Istio installed in the workload clusters.
   • A simple Gloo workspace setup.
2. Install Bookinfo and other sample apps.

Configure outlier detection policies link

You can apply an outlier detection policy at the destination level. For more information, see Applying policies.

Review the following sample configuration file.

  apiVersion: resilience.policy.gloo.solo.io/v2
kind: OutlierDetectionPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: outlier-detection
  namespace: bookinfo
spec:
  applyToDestinations:
  - kind: VIRTUAL_DESTINATION
    selector: {}
  config:
    baseEjectionTime: 30s
    consecutiveErrors: 2
    interval: 1s
    maxEjectionPercent: 100
  

Review the following table to understand this configuration. For more information, see the API docs.

Verify outlier detection policies link

You can test how outlier detection works by opening the Bookinfo app in your browser and observing the reviews app behavior after applying various resources.

1. Create a virtual destination for the reviews app. The virtual destination allows for multicluster routing across clusters for the three different reviews apps in your Bookinfo setup.

     kubectl --context $REMOTE_CONTEXT1 apply -f - <<EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: VirtualDestination
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: reviews-global
     namespace: bookinfo
   spec:
     hosts:
     - reviews.vd
     ports:
     - number: 80
       protocol: HTTP
       targetPort:
         name: http
     services:
     - labels:
         app: reviews
   EOF
     

2. Send a request to the reviews app from the ratings app several times. Notice that you get responses with no stars (v1), black stars (v2), and red stars (v3) from all three reviews apps across clusters.

     kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[].metadata.name}' --context ${REMOTE_CONTEXT1}) -n bookinfo -c ratings --context ${REMOTE_CONTEXT1} -- curl -sS reviews.global:80/reviews/1 -v
     

3. Create the outlier detection policy that you previously reviewed.

     kubectl --context $REMOTE_CONTEXT1 apply -f - <<EOF
   apiVersion: resilience.policy.gloo.solo.io/v2
   kind: OutlierDetectionPolicy
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: outlier-detection
     namespace: bookinfo
   spec:
     applyToDestinations:
     - kind: VIRTUAL_DESTINATION
       selector: {}
     config:
       baseEjectionTime: 30s
       consecutiveErrors: 2
       interval: 1s
       maxEjectionPercent: 100
   EOF
     

4. Repeat the request to the reviews app. Notice that you get responses with no stars (v1) and black stars (v2), but no longer any red stars (v3 in the second cluster). When you apply an outlier detection policy, Gloo enforces locality preference. The request is fulfilled by the reviews apps local to the requesting ratings app when the reviews apps are healthy.

     kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[].metadata.name}' --context ${REMOTE_CONTEXT1}) -n bookinfo -c ratings --context ${REMOTE_CONTEXT1} -- curl -sS reviews.global:80/reviews/1 -v
     

5. Send the reviews v1 app to sleep, to mimic an app failure.

     kubectl --context $REMOTE_CONTEXT1 -n bookinfo patch deploy reviews-v1 --patch '{"spec":{"template":{"spec":{"containers":[{"name":"reviews","command":["sleep","20h"]}]}}}}'
     

6. Repeat the request to the reviews app. Now, you get responses from black stars (v2) and red stars (v3) because reviews v1 is unhealthy and the virtual destination permits load balancing.

     kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[].metadata.name}' --context ${REMOTE_CONTEXT1}) -n bookinfo -c ratings --context ${REMOTE_CONTEXT1} -- curl -sS reviews.global:80/reviews/1 -v
     

7. Remove the sleep command from the reviews v1 app to restore normal behavior.

     kubectl --context $REMOTE_CONTEXT1 -n bookinfo patch deploy reviews-v1 --patch '{"spec":{"template":{"spec":{"containers":[{"name":"reviews","command":[]}]}}}}'
     

8. Keep sending requests to the reviews app. Notice that for the first 30 seconds, you only get responses from black stars (v2) and red stars (v3). After 30 seconds, you also get responses with no stars (v1). The outlier detection policy set the baseEjectionTime to 30s, so the reviews v1 is not returned to the healthy connection pool until after this time elapses.

     kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[].metadata.name}' --context ${REMOTE_CONTEXT1}) -n bookinfo -c ratings --context ${REMOTE_CONTEXT1} -- curl -sS reviews.global:80/reviews/1 -v
     

Cleanup link

  kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete virtualdestination reviews-global
kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete outlierdetectionpolicy outlier-detection