Mirroring

Duplicate outgoing traffic, to test a new app.

Traffic mirroring is also referred to as traffic shadowing. You can send a copy of live traffic to a mirrored service before you deploy your updates to production. This way, you can reduce risks when upgrading your apps by testing out the changes first.

For more information, see the following resources.

Before you begin

  1. Complete the demo setup to install Gloo Mesh, Istio, and Bookinfo in your cluster.

  2. Create the Gloo Mesh resources for this policy in the management and workload clusters.

    The following files are examples only for testing purposes. Your actual setup might vary. You can use the files as a reference for creating your own tests.

    1. Download the following Gloo Mesh resources:
    2. Apply the files to your management cluster.
      kubectl apply -f kubernetes-cluster_gloo-mesh_cluster-1.yaml --context ${MGMT_CONTEXT}
      kubectl apply -f kubernetes-cluster_gloo-mesh_cluster-2.yaml --context ${MGMT_CONTEXT}
      kubectl apply -f workspace_gloo-mesh_anything.yaml --context ${MGMT_CONTEXT}
      
    1. Download the following Gloo Mesh resources:
    2. Apply the files to your workload cluster.
      kubectl apply -f mirror-policy_bookinfo_mirror-policy.yaml --context ${REMOTE_CONTEXT1}
      kubectl apply -f route-table_bookinfo_www-example-com-federated.yaml --context ${REMOTE_CONTEXT1}
      kubectl apply -f virtual-gateway_bookinfo_north-south-gw.yaml --context ${REMOTE_CONTEXT1}
      kubectl apply -f workspace-settings_bookinfo_federated-anything.yaml --context ${REMOTE_CONTEXT1}
      
  3. In order to see the traffic mirroring, you must enable access logging on the remote reviews service in cluster-2. Review the following options to enable access logging.

Configure mirror policies

You can apply a mirror policy at the route level. For more information, see Applying policies.

Review the following sample configuration file.

apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: MirrorPolicy
metadata:
  name: mirror-policy
  namespace: bookinfo
spec:
  applyToRoutes:
  - route:
      labels:
        route: reviews-federated
  config:
    destination:
      port:
        number: 9080
      ref:
        cluster: cluster-1
        name: reviews
        namespace: bookinfo
Review the following table to understand this configuration.
Setting Description
spec.applyToRoutes Configure which routes to apply the policy to, by using labels. The label matches the app and the route from the route table. If omitted, the policy applies to all routes in the workspace.
spec.config.destination.port Specify the port number in the service to mirror traffic to.
spec.config.destination.ref Set the cluster, name, and namespace details for the service that you want to mirror traffic to.
spec.config.percentage Specify the percentage of requests that you want to mirror to the service. The default is 100 to mirror all requests.

Verify mirror policies

  1. Test that your setup currently directs traffic to the reviews service on cluster-2.

    curl -vik --connect-timeout 1 --max-time 5 --resolve www.example.com:32010:127.0.0.1 https://www.example.com:32010/reviews/1
    

    In the output, notice that the color is red, which is a feature only on the reviews app in cluster-2.

    {"id": "1","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"stars": 5, "color": "red"}},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"stars": 4, "color": "red"}}]}
    
  2. Apply the example mirror policy in the cluster with the Bookinfo workspace in your example setup.

    kubectl apply --context ${REMOTE_CONTEXT1} -f mirror-policy_bookinfo_mirror-policy.yaml
    

    Now, 100% of request traffic is still served by the reviews app in cluster-2. Additionally, 100% of the traffic is mirrored to the reviews service in cluster-1. When traffic is mirrored, the requests are sent to the mirrored service with -shadow appended to their Host/Authority headers. For example, cluster-1 becomes cluster-1-shadow.

    These requests are mirrored as “fire and forget,” so the responses are discarded after the request is made. After you make a request to the reviews service, you still see only reviews-v3 responses from cluster-2. However, mirrored traffic still appears in the access logs.

  3. Send another request to the reviews service in cluster-2.

    curl -vik --connect-timeout 1 --max-time 5 --resolve www.example.com:32010:127.0.0.1 https://www.example.com:32010/reviews/1
    
  4. Check the access logs for the reviews service in cluster-2.

    kubectl --context=kind-cluster-2 --namespace bookinfo logs $(kubectl get pod -l app=reviews --context=kind-cluster-2 --namespace bookinfo -o jsonpath={.items..metadata.name}) -c istio-proxy
    

    Example output:

    [2022-03-10T15:42:40.771Z] "GET /reviews/1 HTTP/1.1" 200 - via_upstream - "-" 0 375 31 30 "-" "curl/7.69.1-DEV" "d0df0767-5fe8-4967-8c67-001e14d99434" "reviews.bookinfo.cluster-2.gloo.test.io:9080" "10.244.0.9:9080" inbound|9080|| 127.0.0.6:59309 10.244.0.9:9080 10.244.0.8:36602 outbound_.9080_._.reviews.bookinfo.cluster-2.gloo.test.io default
    
  5. Check the access logs for the reviews service in cluster-1. The access logs are for the mirrored requests that are actually served by the reviews service in cluster-2.

    kubectl --context=kind-cluster-1 --namespace bookinfo logs $(kubectl get pod -l app=reviews,version=v2 --context=kind-cluster-1 --namespace bookinfo -o jsonpath={.items..metadata.name}) -c istio-proxy
    

    Example output:

    [2022-03-10T16:34:09.598Z] "GET /reviews/1 HTTP/1.1" 200 - via_upstream - "-" 0 379 36 36 "-" "curl/7.69.1-DEV" "d25e1c5e-961a-4b14-95c1-3681bb5fbf92" "reviews.bookinfo.cluster-1.gloo.test.io:9080" "10.244.0.9:9080" inbound|9080|| 127.0.0.6:44465 10.244.0.9:9080 10.244.0.7:39180 outbound_.9080_._.reviews.bookinfo.cluster-1.gloo.test.io default