Fault injection
Test the resilience of your apps by injecting delays and connection failures.
Inject faults in a percentage of your requests to test how your app handles the errors. By using the policy, you can avoid deleting pods, delaying packets, or corrupting packets.
You can set two types of faults injection:
- Delays: Delays are timing failures, such as network latency or overloaded upstreams.
- Aborts: Aborts are crash failures, such as HTTP error codes or TCP connection failures.
For more information, see the following resources.
If you import or export resources across workspaces, your policies might not apply. For more information, see Import and export policies.
Before you begin
This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.
Complete the multicluster getting started guide to set up the following testing environment.
- Three clusters along with environment variables for the clusters and their Kubernetes contexts.
- The Gloo
meshctl
CLI, along with other CLI tools such askubectl
andistioctl
. - The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
- Istio installed in the workload clusters.
- A simple Gloo workspace setup.
- Install Bookinfo and other sample apps.
Configure fault injection policies
You can apply a fault injection policy at the route level. For more information, see Applying policies.
Review the following sample configuration files.
Verify fault injection policies
- Create the example fault injection policy for the ratings app.
kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF apiVersion: resilience.policy.gloo.solo.io/v2 kind: FaultInjectionPolicy metadata: annotations: cluster.solo.io/cluster: "" name: faultinjection-basic namespace: bookinfo spec: applyToRoutes: - route: labels: route: ratings config: abort: httpStatus: 418 EOF
Create a route table for the ratings app. Because the policy applies at the route level, Gloo checks for the route in a route table resource.
kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF apiVersion: networking.gloo.solo.io/v2 kind: RouteTable metadata: name: ratings-rt namespace: bookinfo spec: hosts: - ratings http: - forwardTo: destinations: - ref: name: ratings namespace: bookinfo labels: route: ratings workloadSelectors: - {} EOF
Review the following table to understand this configuration. For more information, see the API docs.
Setting Description hosts
The host that the route table routes traffic for. In this example, the ratings
host matches the ratings service within the mesh.http.forwardTo.destinations
The destination to forward requests that come in along the host route. In this example, the ratings service is selected. http.labels
The label for the route. This label must match the label that the policy selects. workloadSelectors
The source workloads within the mesh that this route table routes traffic for. In the example, all workloads are selected. This way, the curl container that you create in subsequent steps can send a request along the ratings route. - Send a request to the ratings app from within the mesh.
- Verify that you notice the fault from the previous examples.
- Abort: All inbound requests to the ratings service result in a
418 Unknown
HTTP status code. - Delay: All inbound requests to the ratings service have a five second delay.
- Both abort and delay: 10% of the calls return
418 Unknown
HTTP status code responses, and 40% have a five second delay before they send a response.
- Abort: All inbound requests to the ratings service result in a
- Optional: Clean up the resources that you created.
kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete routetable ratings-rt kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete faultinjectionpolicy faultinjection-basic