Access logs

Collect and review access logs to monitor the traffic to your ingress gateway proxy.

Leverage the default Envoy access log collector to record logs for the Istio ingress gateway. You can then review these logs to troubleshoot issues as-needed, or scrape these logs to view them in your larger platform logging system. The full record of each request includes source and destination metadata, such as the client ID that initiated the request. Auditors in your organization can use this information to detect malicious activity or unusual amounts of requests to your gateway.

To get started, enable the default Envoy access log collector in your Istio installation. Then, use Gloo access log policies to configure how access logs are recorded for the ingress gateway.

Enable access logging

Enable the default Envoy access log collector when you install or update Istio.

1. Instrument the ingress gateway to produce access logs by updating your Istio installation. The following example meshConfig options configure Envoy to use the default Envoy log format, encode the logs in JSON format, and to store the log files under /dev/stdout. You can add these meshConfig settings to new or existing Istio installations in the following ways:

   • Managed gateway proxies: See the upgrade guide for Istio settings in your Helm chart. In your Helm values file, add the following meshConfig section to your controlPlane installation:

          ...
          istioInstallations:
            controlPlane:
              enabled: true
              installations:
                - istioOperatorSpec:
                    meshConfig:
                      # Enable access logging to /dev/stdout
                      accessLogFile: /dev/stdout
                      # Encoding for the access log (TEXT or JSON). Default value is TEXT.
                      accessLogEncoding: JSON
                      # If empty, the default log format is used.
                      # See the default log format at https://istio.io/latest/docs/tasks/observability/logs/access-log/#default-access-log-format
                      # To change the format, see https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#format-rules
                      accessLogFormat: 
                  revision: auto
            enabled: true
            northSouthGateways:
              - enabled: true
                installations:
                  - gatewayRevision: auto
                    istioOperatorSpec: {}
                name: istio-ingressgateway
           ...
           

   • Manually installed gateway proxies: Follow the manual Istio upgrade guide to add the following section to your Helm values file for the istiod control plane:

     ...
     meshConfig:
       # Enable access logging to /dev/stdout
       accessLogFile: /dev/stdout
       # Encoding for the access log (TEXT or JSON). Default value is TEXT.
       accessLogEncoding: JSON
       # If empty, the default log format is used.
       # See the default log format at https://istio.io/latest/docs/tasks/observability/logs/access-log/#default-access-log-format
       # To change the format, see https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#format-rules
       accessLogFormat: 
     

2. Restart the ingress gateway to start collecting access logs.

   kubectl rollout restart deployment istio-ingressgateway-$REVISION -n gloo-mesh-gateways  
   

3. Optional: Forward access logs to the Gloo telemetry gateway.

View access logs

View the Envoy access logs that are recorded by default for the ingress gateway.

1. Save the external address of the ingress gateway.

   export INGRESS_GW_IP=$(kubectl get svc -n gloo-mesh-gateways istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo $INGRESS_GW_IP
   

2. Generate access logs by sending requests to any route in your Gloo Gateway setup.

   curl -vik http://www.example.com:80/productpage --resolve www.example.com:80:$INGRESS_GW_IP
   

3. View the access logs for the ingress gateway proxy.

   kubectl logs -n gloo-mesh-gateways -l istio=ingressgateway
   

4. Check the logs for the ingress gateway. All access logs for the requests that you previously sent are recorded. For example, the logs might look similar to the following:

   {"authority":"www.example.com","requested_server_name":null,"protocol":"HTTP/1.1","route_name":"productpage-www-example-com.bookinfo.graham0","response_flags":"-","duration":2033,"start_time":"2023-04-20T15:05:31.963Z","request_id":"6aa61345-7c42-42fe-8797-ee31135ba1d9","bytes_sent":4183,"upstream_host":"10.40.1.34:9080","upstream_transport_failure_reason":null,"response_code_details":"via_upstream","upstream_cluster":"outbound|9080||productpage.bookinfo.svc.cluster.local","method":"GET","user_agent":"curl/7.79.1","upstream_local_address":"10.40.0.67:48142","upstream_service_time":"2032","bytes_received":0,"path":"/productpage","x_forwarded_for":"10.40.0.1","downstream_local_address":"10.40.0.67:8080","downstream_remote_address":"10.40.0.1:22305","connection_termination_details":null,"response_code":200}
   

For more information about the default log format, see the Istio default access log docs. To understand each field in the default format, see the Envoy access log docs.

Forward access logs to the Gloo telemetry gateway

By default, the Gloo telemetry pipeline does not collect access logs for the ingress gateway. However, you can configure the telemetry collector agents to pick up the access logs and to forward them to the Gloo telemetry gateway. The telemetry gateway can then forward the logs to a third-party provider, such as Clickhouse. For example, if you use Gloo Portal and you want to monitor Portal analytics, you must configure the Gloo telemetry pipeline to pick up the access logs and forward them to Clickhouse.

To configure the Gloo telemetry pipeline to pick up access logs, follow the steps in Enable Istio access logs in the Gloo OTel pipeline.