Collect and review access logs to monitor the traffic to and from individual services in your service mesh.
Istio leverages the default Envoy access log collector to record logs for each service. You can then review these logs to troubleshoot issues as-needed, or scrape these logs to view them in your larger platform logging system. Digging into the access logs for an app can help you pinpoint issues between particular services in your mesh. Additionally, the full record of each request includes source and destination metadata, such as the client ID that initiated the request. Auditors in your organization can use this information to detect malicious activity or unusual amounts of requests from one app to another, down to the individual workload level.
To get started, enable the default Envoy access log collector in your Istio installation. Then, use Gloo Mesh access log policies to configure how access logs are recorded for your services.
The information in this guide assumes that you have a Gloo Mesh Enterprise license and that you followed the Gloo Mesh getting started guide to install Gloo, install Istio in the workload clusters, and deploy the Bookinfo app.
Enable access logging
Instrument the ingress gateway to produce access logs by updating your Istio installation. The following example
meshConfigoptions configure Envoy to use the default Envoy log format, encode the logs in JSON format, and to store the log files under
/dev/stdout. You can add these
meshConfigsettings to new or existing Istio installations in the following ways:
- Managed gateway proxies: See the upgrade guide for Istio settings in lifecycle manager CRs or in your Helm chart.
- Manually installed gateway proxies: See the manual Istio upgrade guide.
... meshConfig: # Enable access logging to /dev/stdout accessLogFile: /dev/stdout # Encoding for the access log (TEXT or JSON). Default value is TEXT. accessLogEncoding: JSON # If empty, the default log format is used. # See the default log format at https://istio.io/latest/docs/tasks/observability/logs/access-log/#default-access-log-format # To change the format, see https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#format-rules accessLogFormat:
Restart the workloads in your service mesh that you want to collect access logs for. For example, to restart the product page app, run the following command.
kubectl rollout restart deployment productpage-v1 -n bookinfo --context $REMOTE_CONTEXT1
Optional: Forward access logs to the Gloo telemetry gateway.
View access logs
View the Envoy access logs that are recorded by default for workloads in your service mesh.
Note that the following example steps assume that you followed the multicluster getting started guide and Bookinfo sample app guide to install Bookinfo in your workload cluster. However, you can edit these example steps for your own apps as well.
Create a temporary container with the
curlutility in the same namespace as the
kubectl run -it -n bookinfo --context $REMOTE_CONTEXT1 curl \ --image=curlimages/curl:7.73.0 --rm -- sh
From the new terminal, generate access logs by sending requests to
exitwhen you're done.
curl http://ratings:9080/ratings/1 -v
kubectl logs -l app=ratings -c istio-proxy -n bookinfo --context $REMOTE_CONTEXT1
Check the logs for the
ratingsapp. All access logs for the requests that you previously sent are recorded. For example, the logs might look similar to the following:
[2022-06-17T00:29:40.135Z] "GET /ratings/1 HTTP/1.1" 200 - via_upstream - "-" 0 48 2 1 "-" "curl/7.73.0-DEV" "8bdf59f8-0608-4d39-ac73-268519068afb" "ratings:9080" "10.24.1.17:9080" inbound|9080|| 127.0.0.6:59801 10.24.1.17:9080 10.24.0.21:48722 outbound_.9080_._.ratings.bookinfo.svc.cluster.local default [2022-06-17T00:30:08.114Z] "GET /ratings/1 HTTP/1.1" 200 - via_upstream - "-" 0 48 2 2 "-" "curl/7.73.0-DEV" "248926bd-ca16-42a6-866d-d2ab5a1440ee" "ratings:9080" "10.24.1.17:9080" inbound|9080|| 127.0.0.6:47333 10.24.1.17:9080 10.24.0.21:48672 outbound_.9080_._.ratings.bookinfo.svc.cluster.local default
For more information about the default log format, see the Istio default access log docs. To understand each field in the default format, see the Envoy access log docs.
Forward access logs to the Gloo telemetry gateway
By default, the Gloo telemetry pipeline does not collect access logs that you enabled for the workloads in your service mesh. However, you can configure the telemetry collector agents to pick up the access logs and to forward them to the Gloo telemetry gateway. The telemetry gateway can then forward the logs to a third-party provider, such as Clickhouse. For example, if you use the Gloo Gateway Portal feature and you want to monitor Portal analytics, you must configure the Gloo telemetry pipeline to pick up the access logs and forward them to Clickhouse.
To configure the Gloo telemetry pipeline to pick up access logs, follow the steps in Enable Istio access logs in the Gloo OTel pipeline.