Mirroring
Duplicate outgoing traffic to test a new app.
Traffic mirroring is also referred to as traffic shadowing. You can send a copy of live traffic to a mirrored service before you deploy your updates to production. This way, you can reduce risks when upgrading your apps by testing out the changes first.
For more information, see the following resources.
If you import or export resources across workspaces, your policies might not apply. For more information, see Import and export policies.
Before you begin
This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.
Complete the multicluster getting started guide to set up the following testing environment.
- Three clusters along with environment variables for the clusters and their Kubernetes contexts.
- The Gloo
meshctl
CLI, along with other CLI tools such askubectl
andistioctl
. - The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
- Istio installed in the workload clusters.
- A simple Gloo workspace setup.
- Install Bookinfo and other sample apps.
In order to see the traffic mirroring, you must enable access logging on the remote
reviews-v3
service incluster2
.
Configure mirror policies
You can apply a mirror policy at the route level. For more information, see Applying policies.
You cannot apply this policy to a route that already has a redirect, rewrite, or direct response action. Keep in mind that these actions might not be explicitly defined in the route configuration. For example, invalid routes are automatically replaced with a direct response action, such as when the backing destination is wrong. First, verify that your route configuration is correct. Then, decide whether to apply the policy. To apply the policy, remove any redirect, rewrite, or direct response actions. To keep the actions and not apply the policy, change the route labels of either the policy or the route.
Review the following sample configuration file.
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: MirrorPolicy
metadata:
annotations:
cluster.solo.io/cluster: ""
name: mirror-policy
namespace: bookinfo
spec:
applyToRoutes:
- route:
labels:
route: reviews-federated
config:
destination:
port:
number: 9080
ref:
cluster: cluster-1
name: reviews
namespace: bookinfo
Review the following table to understand this configuration. For more information, see the API docs.
Setting | Description |
---|---|
spec.applyToRoutes | Use labels to configure which routes to apply the policy to. This example label matches the app and route from the example route table that you apply separately. If omitted and you do not have another selector such as applyToDestinations , the policy applies to all routes in the workspace. |
spec.config.destination.port | Specify the port number in the service to mirror traffic to. |
spec.config.destination.ref | Set the cluster, name, and namespace details for the service that you want to mirror traffic to. |
spec.config.percentage | Specify the percentage of requests that you want to mirror to the service. The default is 100 to mirror all requests. |
Verify mirror policies
Verify a mirror policy by checking for mirrored logs.
Apply the example mirror policy in the cluster with the Bookinfo workspace in your example setup.
kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF apiVersion: trafficcontrol.policy.gloo.solo.io/v2 kind: MirrorPolicy metadata: annotations: cluster.solo.io/cluster: "" name: mirror-policy namespace: bookinfo spec: applyToRoutes: - route: labels: route: reviews-federated config: destination: port: number: 9080 ref: cluster: cluster-1 name: reviews namespace: bookinfo EOF
Create a route table for the reviews app. Because mirror policies apply at the route level, Gloo checks for the route in a route table resource.
kubectl apply --context ${REMOTE_CONTEXT1} -f - << EOF apiVersion: networking.gloo.solo.io/v2 kind: RouteTable metadata: name: reviews-rt namespace: bookinfo spec: hosts: - reviews http: - forwardTo: destinations: - ref: name: reviews namespace: bookinfo cluster: ${REMOTE_CLUSTER1} labels: route: reviews-federated workloadSelectors: - {} EOF
Review the following table to understand this configuration. For more information, see the API docs.
Setting Description hosts
The host that the route table routes traffic for. In this example, the ratings
host matches the ratings service within the mesh.http.forwardTo.destinations
The destination to forward requests that come in along the host route. In this example, the ratings service is selected. http.labels
The label for the route. This label must match the label that the policy selects. workloadSelectors
The source workloads within the mesh that this route table routes traffic for. In the example, all workloads are selected. This way, the curl container that you create in subsequent steps can send a request along the ratings route. Send a request to the
reviews-v1
app. Create a temporary curl pod in thebookinfo
namespace, so that you can test the app setup. You can also use this method in Kubernetes 1.23 or later, but an ephemeral container might be simpler.- Create the curl pod.
kubectl run -it -n bookinfo --context $REMOTE_CONTEXT1 curl \ --image=curlimages/curl:7.73.0 --rm -- sh
- Send a request to the ratings app.
curl -v http://reviews:9080/reviews/1
- Exit the temporary pod. The pod deletes itself.
exit
In the output, notice that no stars are included. Reviews v1 does not include stars in the reviews, whereas reviews v2 includes black stars.
{"id": "1","podname": "reviews-v2-858f99c99-dll8j","clustername": "null","reviews": [{ "reviewer": "Reviewer1", "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!", "rating": {"error": "Ratings service is currently unavailable"}},{ "reviewer": "Reviewer2", "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare.", "rating": {"error": "Ratings service is currently unavailable"}}]}
- Create the curl pod.
Check the pod logs for the reviews v2 app. The access logs are for the mirrored requests that are actually served by the reviews v1 app.
kubectl logs deploy/reviews-v2 -c istio-proxy -n bookinfo --context $REMOTE_CONTEXT1
Example output:
{ "x_forwarded_for": null, "duration": 1611, "downstream_remote_address": "10.244.2.47:43102", "authority": "reviews:9080", "bytes_received": 0, "bytes_sent": 487, "upstream_cluster": "inbound|9080||", "path": "/reviews/1", "requested_server_name": "outbound_.9080_._.reviews.bookinfo.svc.cluster.local", "protocol": "HTTP/1.1", "upstream_host": "10.244.1.16:9080", "method": "GET", "route_name": "default", "upstream_service_time": "1609", "response_code": 200, "response_code_details": "via_upstream", "upstream_transport_failure_reason": null, "connection_termination_details": null, "start_time": "2022-09-06T16:08:00.510Z", "downstream_local_address": "10.244.1.16:9080", "user_agent": "curl/7.73.0-DEV", "response_flags": "-", "upstream_local_address": "127.0.0.6:45009", "request_id": "b6e1f09c-b029-49e0-95fe-8affaaf2f542" }
Cleanup
You can optionally remove the resources that you set up as part of this guide.
kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete RouteTable reviews-rt
kubectl --context $REMOTE_CONTEXT1 -n bookinfo delete MirrorPolicy mirror-policy