Apply fault injection

Apply a fault injection policy to the ratings service to delay requests and simulate network issues or an overloaded service. A delay simulates an overloaded upstream service or network issues, and can help you build more resilient apps.

  1. Verify that you can successfully send requests to the ratings app from within the mesh.

  2. Create a fault injection policy to delay responses from the ratings app by 10 seconds, and a route table specifically for testing access to the ratings app.

      kubectl apply --context $MGMT_CONTEXT -f- <<EOF
    apiVersion: resilience.policy.gloo.solo.io/v2
    kind: FaultInjectionPolicy
    metadata:
      name: faultinjection-basic-delay
      namespace: bookinfo
    spec:
      applyToRoutes:
        - route:
            labels:
              route: ratings
      config:
        delay:
          fixedDelay: 10s
    ---
    apiVersion: networking.gloo.solo.io/v2
    kind: RouteTable
    metadata:
      name: ratings-rt
      namespace: bookinfo
    spec:
      hosts:
      - ratings
      http:
      - forwardTo:
          destinations:
          - ref:
              name: ratings
              namespace: bookinfo
        labels:
          route: ratings
      workloadSelectors:
      - {}
    EOF
      
  3. Send another request to the ratings app by using the same method as in step 1. Note that this time, the app’s response is delayed due to the fault injection.

Explore the UI

Use the Gloo UI to evaluate the health and efficiency of your service mesh. You can review the analysis and insights for your service mesh, such as recommendations to harden your Istio environment and steps to implement them in your environment.

Launch the dashboard

  1. Open the Gloo UI. The Gloo UI is served from the gloo-mesh-ui service on port 8090. You can connect by using the meshctl or kubectl CLIs.

  2. Review your Dashboard for an at-a-glance overview of your Gloo Mesh Enterprise environment. Environment insights, health, status, inventories, security, and more are summarized in the following cards:

    • Analysis and Insights: Gloo Mesh Enterprise recommendations for how to improve your Istio setups, and if installed, your Cilium setups.
    • Gloo, Istio, and Cilium health: A status check of the Gloo Mesh Enterprise, Istio, and if installed, Cilium installations in each cluster.
    • Certificates Expiry: Validity timelines for your root and intermediate Istio certificates.
    • Cluster Services: Inventory of services across all clusters in your Gloo Mesh Enterprise setup, and whether those services are in a service mesh or not.
    • Istio FIPS: FIPS compliance checks for the istiod control planes and Istio data plane workloads.
    • Zero Trust: Number of service mesh workloads that receive only mutual TLS (mTLS)-encrypted traffic, and number of external services that are accessed from the mesh.
    Figure: Gloo UI dashboard
    Figure: Gloo UI dashboard
  3. If you installed the Cilium CNI, click the Cilium tab in the Gloo, Istio, and Cilium health card. Verify that the Cilium version in each cluster was discovered.

    Cilium installation health card in the Gloo UI

Check insights

Review the insights for your environment. Gloo Mesh Enterprise comes with an insights engine that automatically analyzes your Istio and Cilium setups for health issues. Then, Gloo shares these issues along with recommendations to harden your Istio and Cilium setups. The insights give you a checklist to address issues that might otherwise be hard to detect across your environment.

  1. On the Analysis and Insights card of the dashboard, you can quickly see a summary of the insights for your environment, including how many insights are available at each severity level, and the type of insight.

    Figure: Insights and analysis card
    Figure: Insights and analysis card

  2. View the list of insights by clicking the Details button, or go to the Insights page.

  3. On the Insights page, you can view recommendations to harden your Istio, and if installed, Cilium setup, and steps to implement them in your environment. Gloo Mesh Enterprise analyzes your setup, and returns individual insights that contain information about errors and warnings in your environment, best practices you can use to improve your configuration and security, and more.

    Figure: Insights page
    Figure: Insights page

  4. On an insight that you want to resolve, click Details. The details modal shows more data about the insight, such as the time when it was last observed in your environment, and if applicable, the extended settings or configuration that the insight applies to.

    Figure: Example insight
    Figure: Example insight
  5. Click the Target YAML tab to see the resource file that the insight references, and click the View Resolution Steps tab to see guidance such as steps for fixing warnings and errors in your resource configuration or recommendations for improving your security and setup.

Optional: Apply a Cilium network policy

If you installed the Solo distribution of the Cilium CNI, deploy a demo app to visualize Cilium network traffic in the Gloo UI, and try out a Cilium network policy to secure and control traffic flows between app microservices.

  1. Deploy the Cilium Star Wars demo app in your cluster.

    1. Create a namespace for the demo app, and include the starwars services in your service mesh.

        kubectl create ns starwars
      kubectl label ns starwars istio-injection=enabled
        
    2. Deploy the demo app, which includes tiefighter, xwing, and deathstar pods, and a deathstar service. The tiefighter and deathstar pods have the org=empire label, and the xwing pod has the org=alliance label.

        kubectl -n starwars apply -f https://raw.githubusercontent.com/cilium/cilium/$CILIUM_VERSION/examples/minikube/http-sw-app.yaml
        
    3. Verify that the demo pods and service are running.

        kubectl get pods,svc -n starwars
        

      Example output:

        NAME                             READY   STATUS    RESTARTS   AGE
      pod/deathstar-6fb5694d48-5hmds   1/1     Running   0          107s
      pod/deathstar-6fb5694d48-fhf65   1/1     Running   0          107s
      pod/tiefighter                   1/1     Running   0          107s
      pod/xwing                        1/1     Running   0          107s
      
      NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
      service/deathstar    ClusterIP   10.96.110.8   <none>        80/TCP    107s
        
  2. Generate some network traffic by sending requests from the xwing and tiefighter pods to the deathstar service.

      kubectl exec xwing -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing
    kubectl exec tiefighter -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing
      

    Example output for both commands:

      Ship landed
    Ship landed
      
  3. View network traffic information in the Gloo UI.

    1. Open the Gloo UI.

      • meshctl:
          meshctl dashboard
          
      • kubectl:
        1. Port-forward the gloo-mesh-ui service on 8090.
            kubectl port-forward -n gloo-mesh svc/gloo-mesh-ui 8090:8090
            
        2. Open your browser and connect to http://localhost:8090.
    2. From the left-hand navigation, click Observability > Graph.

    3. View the network graph for the Star Wars app. The graph is automatically generated based on which apps talk to each other.

      Figure: Gloo UI network graph for the Star Wars app
      Figure: Gloo UI network graph for the Star Wars app
  4. Create a Cilium network policy that allows only apps that have the org=empire label to access the deathstar app. After you create this access policy, only the tiefighter pod can access the deathstar app.

      kubectl apply -f - << EOF
    apiVersion: "cilium.io/v2"
    kind: CiliumNetworkPolicy
    metadata:
      name: "rule1"
      namespace: starwars
    spec:
      description: "L3-L4 policy to restrict deathstar access to empire ships only"
      endpointSelector:
        matchLabels:
          org: empire
          class: deathstar
      ingress:
      - fromEndpoints:
        - matchLabels:
            org: empire
        toPorts:
        - ports:
          - port: "80"
            protocol: TCP
    EOF
      
  5. Send another request from the tiefighter pod to the deathstar service.

      kubectl exec tiefighter -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing
      

    The request succeeds, because requests from apps with the org=empire label are permitted. Example output:

      Ship landed
      
  6. Send another request from the xwing pod to the deathstar service.

      kubectl exec xwing -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing
      

    This request hangs, because requests from apps without the org=empire label are not permitted. No Layer 7 HTTP response code is returned, because the request is dropped on Layer 3. You can enter control+c to stop the curl request, or wait for it to time out.

  7. You can also check the metrics to verify that the policy allowed or blocked requests.

    1. Open the Prometheus expression browser.
      • meshctl: For more information, see the CLI documentation.
          meshctl proxy prometheus
          
      • kubectl:
        1. Port-forward the prometheus-server deployment on 9091.
            kubectl -n gloo-mesh port-forward deploy/prometheus-server 9091
            
        2. Open your browser and connect to localhost:9091/.
    2. In the Expression search bar, paste the following query and click Execute.
        rate(hubble_drop_total{destination_workload_id=~"deathstar.+"}[5m])
        
    3. Verify that you can see requests from the xwing pod to the deathstar service that were dropped because of the network policy.

Next steps

Now that you have Gloo Mesh Enterprise and Istio up and running, check out some of the following resources to learn more about Gloo Mesh and expand your service mesh capabilities.

Gloo Mesh Enterprise:

Istio: Now that you have Gloo Mesh Enterprise and Istio installed, you can use Gloo to manage your Istio service mesh resources. You don’t need to directly configure any Istio resources going forward.

Cilium: If you installed the Solo distribution of the Cilium CNI in your clusters:

Help and support:

Cleanup

You can optionally remove the resources that you set up as part of this guide.

  1. Delete the fault injection policy and testing route table.

      kubectl delete FaultInjectionPolicy faultinjection-basic-delay -n bookinfo --context $MGMT_CONTEXT
    kubectl delete RouteTable ratings-rt -n bookinfo --context $MGMT_CONTEXT
      
  2. If you installed the Solo distribution of the Cilium CNI, remove the demo app resources and namespace.

      kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/$CILIUM_VERSION/examples/minikube/http-sw-app.yaml -n starwars
    kubectl delete cnp rule1 -n starwars
    kubectl delete ns starwars
      
  3. If you no longer need this quick-start Gloo Mesh environment, you can follow the steps in the uninstall guide.