Mirror traffic

Traffic mirroring is also referred to as traffic shadowing. You can send a copy of live traffic to a mirrored service before you deploy your updates to production. This way, you can reduce risks when upgrading your apps by testing out the changes first.

For more information, see the following resources.

• Istio docs for mirroring
• Gloo Mesh API docs

Before you begin link

1. Complete the multicluster getting started guide to set up the following testing environment.

   • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
   • The Gloo meshctl CLI, along with other CLI tools such as kubectl and istioctl.
   • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
   • Istio installed in the workload clusters.
   • A simple Gloo workspace setup.
2. Install Bookinfo and other sample apps.

3. In order to see the traffic mirroring, you must enable access logging on the remote reviews-v3 service in cluster2.

Configure mirror policies link

You can apply a mirror policy at the route level. For more information, see Applying policies.

Review the following sample configuration file.

  apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: MirrorPolicy
metadata:
  annotations:
    cluster.solo.io/cluster: ""
  name: mirror-policy
  namespace: bookinfo
spec:
  applyToRoutes:
  - route:
      labels:
        route: reviews-federated
  config:
    destination:
      port:
        number: 9080
      ref:
        cluster: cluster-1
        name: reviews
        namespace: bookinfo
  

Review the following table to understand this configuration. For more information, see the API docs.

Verify mirror policies link

1. Apply the example mirror policy in the cluster with the Bookinfo workspace in your example setup.

      kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF
      apiVersion: trafficcontrol.policy.gloo.solo.io/v2
      kind: MirrorPolicy
      metadata:
        annotations:
          cluster.solo.io/cluster: ""
        name: mirror-policy
        namespace: bookinfo
      spec:
        applyToRoutes:
        - route:
            labels:
              route: reviews-federated
        config:
          destination:
            port:
              number: 9080
            ref:
              cluster: cluster-1
              name: reviews
              namespace: bookinfo
      EOF
      

2. Create a route table for the reviews app. Because mirror policies apply at the route level, Gloo checks for the route in a route table resource.

     kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF
   apiVersion: networking.gloo.solo.io/v2
   kind: RouteTable
   metadata:
     name: reviews-rt
     namespace: bookinfo
   spec:
     hosts:
     - reviews
     http:
     - forwardTo:
         destinations:
         - ref:
             name: reviews
             namespace: bookinfo
             cluster: ${REMOTE_CLUSTER1}
       labels:
         route: reviews-federated
     workloadSelectors:
     - {}
   EOF
     

   Review the following table to understand this configuration. For more information, see the API docs.

   Setting	Description
   hosts	The host that the route table routes traffic for. In this example, the ratings host matches the ratings service within the mesh.
   http.forwardTo.destinations	The destination to forward requests that come in along the host route. In this example, the ratings service is selected.
   http.labels	The label for the route. This label must match the label that the policy selects.
   workloadSelectors	The source workloads within the mesh that this route table routes traffic for. In the example, all workloads are selected. This way, the curl container that you create in subsequent steps can send a request along the ratings route.

3. Send a request to the reviews-v1 app.

   Use the kubectl debug command to create an ephemeral curl container in the deployment. This way, the curl container inherits any permissions from the app that you want to test. If you don’t run Kubernetes 1.23 or later, you can deploy a separate curl pod or manually add the curl container as shown in the other tab.

     kubectl --context ${REMOTE_CONTEXT1} -n bookinfo debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=reviews -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -v http://reviews:9080/reviews/1
     

   If the output has an error about EphemeralContainers, see Ephemeral containers don’t work when testing Bookinfo.

   Create a temporary curl pod in the bookinfo namespace, so that you can test the app setup. You can also use this method in Kubernetes 1.23 or later, but an ephemeral container might be simpler, as shown in the other tab.

   1. Create the curl pod.

        kubectl run -it -n bookinfo --context $REMOTE_CONTEXT1 curl \
        --image=curlimages/curl:7.73.0 --rm  -- sh
        

   2. Send a request to the ratings app.

        curl -v http://reviews:9080/reviews/1
        

   3. Exit the temporary pod. The pod deletes itself.

        exit
        

   In the output, notice that no stars are included. Reviews v1 does not include stars in the reviews, whereas reviews v2 includes black stars.

     {"id": "1","podname": "reviews-v2-858f99c99-dll8j","clustername": "null","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"error": "Ratings service is currently unavailable"}},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"error": "Ratings service is currently unavailable"}}]}
     

4. Check the pod logs for the reviews v2 app. The access logs are for the mirrored requests that are actually served by the reviews v1 app.

     kubectl logs deploy/reviews-v2 -c istio-proxy -n bookinfo --context $REMOTE_CONTEXT1
     

   Example output:

      {
        "x_forwarded_for": null,
        "duration": 1611,
        "downstream_remote_address": "10.244.2.47:43102",
        "authority": "reviews:9080",
        "bytes_received": 0,
        "bytes_sent": 487,
        "upstream_cluster": "inbound|9080||",
        "path": "/reviews/1",
        "requested_server_name": "outbound_.9080_._.reviews.bookinfo.svc.cluster.local",
        "protocol": "HTTP/1.1",
        "upstream_host": "10.244.1.16:9080",
        "method": "GET",
        "route_name": "default",
        "upstream_service_time": "1609",
        "response_code": 200,
        "response_code_details": "via_upstream",
        "upstream_transport_failure_reason": null,
        "connection_termination_details": null,
        "start_time": "2022-09-06T16:08:00.510Z",
        "downstream_local_address": "10.244.1.16:9080",
        "user_agent": "curl/7.73.0-DEV",
        "response_flags": "-",
        "upstream_local_address": "127.0.0.6:45009",
        "request_id": "b6e1f09c-b029-49e0-95fe-8affaaf2f542"
      }
      

5. Optional: Clean up the resources that you created.

     kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete RouteTable reviews-rt
   kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete MirrorPolicy mirror-policy