Fault injection

Inject faults in a percentage of your requests to test how your app handles the errors. By using the policy, you can avoid deleting pods, delaying packets, or corrupting packets.

You can set two types of faults injection:

• Delays: Delays are timing failures, such as network latency or overloaded upstreams.
• Aborts: Aborts are crash failures, such as HTTP error codes or TCP connection failures.

For more information, see the following resources.

• Istio docs for fault injection
• Gloo Mesh Gateway API docs

Before you begin link

1. Set up Gloo Mesh Gateway in a single cluster.
2. Install Bookinfo and other sample apps.

3. Configure an HTTP listener on your gateway and set up basic routing for the sample apps.

4. Get the external address of your ingress gateway. The steps vary depending on the type of load balancer that backs the ingress gateway.

• LoadBalancer IP address:

    export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  echo $INGRESS_GW_IP
    

• LoadBalancer hostname:

    export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
  echo $INGRESS_GW_IP
    

Note: Depending on your environment, you might see <pending> instead of an external IP address. For example, if you are testing locally in kind or minikube, or if you have insufficent permissions in your cloud platform, you can instead port-forward the service port of the ingress gateway:

  kubectl -n gloo-mesh-gateways port-forward deploy/istio-ingressgateway-1-18 8081
  

Configure fault injection policies link

You can apply a fault injection policy at the route level. For more information, see Applying policies.

Abort example link

The following example is for a simple fault injection abort policy that returns a 418 HTTP response. No delay is configured.

  apiVersion: resilience.policy.gloo.solo.io/v2
kind: FaultInjectionPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: faultinjection-basic
  namespace: bookinfo
spec:
  applyToRoutes:
  - route:
      labels:
        route: ratings
  config:
    abort:
      httpStatus: 418
  

Review the following table to understand this configuration. For more information, see the API docs.

Delay example link

The following example is for a simple fault injection delay policy with a default value for the percentage. No abort is configured.

  apiVersion: resilience.policy.gloo.solo.io/v2
kind: FaultInjectionPolicy
metadata:
  name: faultinjection-basic-delay
  namespace: bookinfo
spec:
  applyToRoutes:
    - route:
        labels:
          route: ratings
  config:
    delay:
      fixedDelay: 5s
  

Review the following table to understand this configuration. For more information, see the API docs.

Abort and delay example link

The following example is for a fault injection policy that both delays and aborts requests. Delays and aborts are independent of one another. When both values are set, your requests are either delayed only, delayed and aborted, or aborted only.

  apiVersion: resilience.policy.gloo.solo.io/v2
kind: FaultInjectionPolicy
metadata:
  name: faultinjection-basic-abort-and-delay
  namespace: bookinfo
  annotations:
    cluster.solo.io/cluster: $REMOTE_CLUSTER1
spec:
  applyToRoutes:
    - route:
        labels:
          route: ratings
  config:
    abort:
      httpStatus: 418
      percentage: 10
    delay:
      percentage: 40
      fixedDelay: 5s
  

Review the following table to understand this configuration. For more information, see the API docs.

Verify fault injection policies link

1. Create the example fault injection policy for the ratings app.

     kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF
   apiVersion: resilience.policy.gloo.solo.io/v2
   kind: FaultInjectionPolicy
   metadata:
     annotations:
       cluster.solo.io/cluster: ""
     name: faultinjection-basic
     namespace: bookinfo
   spec:
     applyToRoutes:
     - route:
         labels:
           route: ratings
     config:
       abort:
         httpStatus: 418
   EOF
     

2. Send a request to the app.
   • HTTP:

       curl -vik --resolve www.example.com:80:${INGRESS_GW_IP} http://www.example.com:80/ratings/1 
       

   • HTTPS:

       curl -vik --resolve www.example.com:443:${INGRESS_GW_IP} https://www.example.com:443/ratings/1 
       

3. Verify that you notice the fault from the previous examples.
   • Abort: All inbound requests to the ratings service result in a 418 Unknown HTTP status code.
   • Delay: All inbound requests to the ratings service have a five second delay.
   • Both abort and delay: 10% of the calls return 418 Unknown HTTP status code responses, and 40% have a five second delay before they send a response.

Cleanup link

  kubectl -n bookinfo delete RouteTable ratings-rt
kubectl -n bookinfo delete FaultInjectionPolicy faultinjection-basic