Block egress traffic with an egress gateway
Use an egress gateway to allow egress traffic to external endpoints for only certain services in the mesh.
Route all egress traffic for an external service through an egress gateway and ensure that only certain services in the mesh can send requests to the external endpoint that it serves. All other services in the mesh are blocked from sending requests to the external endpoint. You can also use the egress traffic to perform TLS origination and secure the connection to the external endpoint. For more information, see Perform TLS origination with egress gateways.
Blocking egress traffic with an egress gateway and access policies works only if clients send traffic to the egress gateway through the client’s sidecar proxy. If the sidecar proxy is bypassed, requests to an external endpoint might still be allowed, which can be a risk for malicious attacks. To make sure that no egress traffic is allowed, even if the client’s sidecar proxy is bypassed, create a network policy that blocks all egress traffic in addition to setting up an egress gateway. You can further reduce the surface for malicious attacks by deploying the egress gateway to a dedicated node. Also note that you cannot apply other policies, such as outlier detection or active healthchecks, to the egress gateway.
Before you begin
Set the names of your clusters from your infrastructure provider. If your clusters have different names, specify those names instead.
export REMOTE_CLUSTER1=<cluster1> export REMOTE_CLUSTER2=<cluster2> ...
- Save the kubeconfig contexts for your clusters. Run
kubectl config get-contexts
, look for your cluster in theCLUSTER
column, and get the context name in theNAME
column. Note: Do not use context names with underscores. The generated certificate that connects workload clusters to the management cluster uses the context name as a SAN specification, and underscores in SAN are not FQDN compliant. You can rename a context by runningkubectl config rename-context "<oldcontext>" <newcontext>
.export MGMT_CONTEXT=<management-cluster-context> export REMOTE_CONTEXT1=<remote-cluster1-context> export REMOTE_CONTEXT2=<remote-cluster2-context> ...
Decide on the outbound traffic policy for your service mesh. If you followed the steps to install Istio with the Istio lifecycle manager, the outbound traffic policy for your service mesh is set to
ALLOW_ANY
by default and allows your services in the mesh to reach any external endpoint. You can change this setting and instead block all egress traffic to hosts that are not part of your service mesh by changing the outbound traffic policy toREGISTRY_ONLY
as shown in the following steps. Note that if you manually installed Istio, follow the steps in Upgrade Istio to update that setting.- Open the
gm-istiod-values.yaml
file that you used to create the istiod control plane. - Change the
outboundTrafficPolicy
field toREGISTRY_ONLY
. - Apply your changes by using the following command.
kubectl apply -f gm-istiod-values.yaml --context $MGMT_CONTEXT
- Open the
Review Supported versions to choose the Solo distribution of Istio that you want to use, and save the version information in the following environment variables.
REPO
: The repo key for the Solo distribution of Istio that you can get by logging in to the Support Center and reviewing the Istio images built by Solo.io support article.ISTIO_IMAGE
: The version that you want to use with thesolo
tag, such as1.22.3-patch0-solo
. You can optionally append other tags of Solo distributions of Istio as needed.REVISION
: Take the Istio major and minor versions and replace the periods with hyphens, such as1-22
.
For testing environments only, you can deploy a revisionless installation. Revisionless installations permit in-place upgrades, which are quicker than the canary-based upgrades that revisioned installations require. To omit a revision, do not set a revision environment variable. Then in the following sections, you edit the sampleIstioLifecycleManager
andGatewayLifecycleManager
files that you download to remove therevision
andgatewayRevision
fields. Note that if you deploy multiple Istio installations in the same cluster, only one installation can be revisionless.export REPO=<repo-key> export ISTIO_IMAGE=1.22.3-patch0-solo export REVISION=1-22
Istio 1.21: In Istio 1.21, the VERIFY_CERTIFICATE_AT_CLIENT
feature flag is set to true by default to automatically verify server certificates by using the OS CA certificates on the cluster node.
Block egress traffic with an egress gateway
Prepare a
GatewayLifecycleManager
to deploy an egress gateway.Download the
gm-egress-gateway.yaml
example file.curl -0L https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/gloo-mesh/istio-install/gm-managed/gm-egress-gateway.yaml > gm-egress-gateway.yaml
Update the example file with the environment variables that you previously set, and optionally further edit the file to provide your own details. Save the updated file as
gm-egress-gateway-values.yaml
. For more information, see the API reference.- Tip: Instead of updating the file manually, try running a terminal command to substitute values, such as the following command.
envsubst < gm-egress-gateway.yaml > gm-egress-gateway-values.yaml open gm-egress-gateway-values.yaml
- Tip: Instead of updating the file manually, try running a terminal command to substitute values, such as the following command.
Apply the
GatewayLifecycleManager
resource to your management cluster.kubectl apply -f gm-egress-gateway-values.yaml --context $MGMT_CONTEXT
Verify that the egress gateway is deployed in your workload clusters.
kubectl get pods -n gloo-mesh-gateways --context $REMOTE_CONTEXT1 kubectl get pods -n gloo-mesh-gateways --context $REMOTE_CONTEXT2
Log in to the reviews app and verify that you cannot reach
www.google.com
.kubectl --context ${REMOTE_CONTEXT1} -n bookinfo debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=reviews -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -vik www.google.com
Example output:
* Mark bundle as not supporting multiuse < HTTP/1.1 502 Bad Gateway HTTP/1.1 502 Bad Gateway < date: Thu, 11 May 2023 17:07:43 GMT date: Thu, 11 May 2023 17:07:43 GMT < server: envoy server: envoy < content-length: 0 content-length: 0
Create the
global
namespace in one of your workload clusters.kubectl create namespace global --context $REMOTE_CONTEXT1
Create a virtual gateway resource to configure your egress gateway going forward.
kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF apiVersion: networking.gloo.solo.io/v2 kind: VirtualGateway metadata: annotations: cluster.solo.io/cluster: "" labels: app: istio-egressgateway istio: egressgateway name: egress-vg namespace: gloo-mesh-gateways spec: listeners: - http: {} appProtocol: HTTPS exposedExternalServices: - host: www.google.com port: number: 443 tls: mode: ISTIO_MUTUAL workloads: - selector: labels: app: istio-egressgateway istio: egressgateway EOF
Setting Description spec.listeners.exposedExternalServices.host
Select the external services that you want to serve on this egress gateway. To select the external services, you define the hostname that must be configured in the external service. spec.listeners.port
Enter the port number that services in the mesh use in combination with the host
to send a request to the egress gateway.spec.listeners.tls
Select the TLS mode that you want to use to connect to the egress gateway from within the mesh. In this example, requests for the www.google.com
host are rerouted to the egress gateway first by using mutual TLS. The incoming TLS connection is terminated at the egress gateway.Create an external service for
www.google.com
.kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF apiVersion: networking.gloo.solo.io/v2 kind: ExternalService metadata: name: google namespace: global spec: hosts: - "www.google.com" ports: - clientsideTls: {} egressGatewayRoutes: portMatch: 80 virtualGatewayRefs: - cluster: $REMOTE_CLUSTER1 name: egress-vg namespace: gloo-mesh-gateways - cluster: $REMOTE_CLUSTER2 name: egress-vg namespace: gloo-mesh-gateways name: https number: 443 protocol: HTTPS EOF
Setting Description spec.hosts
Enter the external hostname that you want to allow access to for services in the mesh. spec.ports.clientsideTls
Allow TLS origination to the www.google.com
endpoint.spec.ports.egressGatewayRoutes.virtualGatewayRefs
Enter the details for your egress gateway. Make sure to enter the egress gateway details for both of your workload clusters. spec.ports.egressGatewayRoutes.portMatch
Select the port that clients use when sending a request to the external endpoint. Note that this port does not represent the port that the client’s sidecar proxy uses to connect to the egress gateway. Instead, make sure to enter the source port that is used in the original client request. spec.ports.name
Enter the port number that you want to use to connect to the external endpoint. When a client request is sent to the gateway, the client’s sidecar proxy intercepts this traffic and securely connects to the egress gateway via mutual TLS. The egress gateway then terminates the incoming TLS connection. Because this example defines an HTTPS port, a new TLS connection is originated at the egress gateway to securely connect to the www.google.com
external endpoint.Verify that you can access
www.google.com
from the reviews app.kubectl --context ${REMOTE_CONTEXT1} -n bookinfo debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=reviews -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -vik www.google.com
Example output:
* Connected to www.google.com (240.240.59.19) port 80 (#0) > GET / HTTP/1.1 > Host: www.google.com > User-Agent: curl/8.1.1-DEV > Accept: */* > < HTTP/1.1 200 OK < date: Wed, 24 May 2023 20:23:50 GMT < expires: -1 < cache-control: private, max-age=0 < content-type: text/html; charset=ISO-8859-1 < content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-k4p_jp_okP7dor-UgJJt5Q' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp < p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info." < server: envoy HTTP/1.1 200 OK ...
Verify that traffic was routed through the egress gateway by looking at the egress gateway logs.
kubectl logs $(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=istio-egressgateway -A -o jsonpath='{.items[0].metadata.name}') -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
Example output:
{"upstream_service_time":"55","response_code":200,"response_flags":"-","authority":"www.google.com","bytes_received":0,"duration":55,"method":"GET","request_id":"f0d0412a-a393-9190-a49b-23baaadf7d22","route_name":null,"requested_server_name":"www.google.com","upstream_host":"142.251.161.105:443","downstream_remote_address":"10.84.1.21:52038","downstream_local_address":"10.84.0.24:8443","start_time":"2023-05-24T20:16:57.753Z","bytes_sent":17238,"response_code_details":"via_upstream","upstream_cluster":"outbound|443||www.google.com","upstream_transport_failure_reason":null,"path":"/","x_forwarded_for":"10.84.1.21","user_agent":"curl/8.1.1-DEV","upstream_local_address":"10.84.0.24:57268","connection_termination_details":null,"protocol":"HTTP/1.1"}
Get the virtual service that was created for your external service and make sure that it lists port 443 for the
www.google.com
destination. This setting ensures the egress gateway originates a new TLS connection before connecting towww.google.com
.kubectl get virtualservices -n gloo-mesh-gateways --context $REMOTE_CONTEXT1 | grep externalservice-google
kubectl get virtualservice <virtual-service-name> -n gloo-mesh-gateways --context $REMOTE_CONTEXT1 -o yaml
Example output:
... spec: exportTo: - bookinfo - default - global - gloo-mesh - gloo-mesh-gateways - gm-iop-1-22 - helloworld - httpbin - istio-system gateways: - virtualgateway-egress-vg-gloo-m-ef7dba3d044465c79ef179255be34f5 - mesh hosts: - www.google.com http: - match: - gateways: - virtualgateway-egress-vg-gloo-m-ef7dba3d044465c79ef179255be34f5 port: 8443 route: - destination: host: www.google.com port: number: 443 ...
Create an access policy for your external service to allow only the product page app to access the
www.google.com
external endpoint.kubectl apply --context $REMOTE_CONTEXT1 -f- <<EOF apiVersion: security.policy.gloo.solo.io/v2 kind: AccessPolicy metadata: annotations: cluster.solo.io/cluster: "" name: egress-access-policy namespace: gloo-mesh-gateways spec: applyToDestinations: - kind: EXTERNAL_SERVICE port: number: 443 selector: cluster: $REMOTE_CLUSTER1 name: google namespace: global config: authz: allowedClients: - serviceAccountSelector: cluster: $REMOTE_CLUSTER1 labels: account: productpage namespace: bookinfo EOF
Send a request to
www.google.com
from the reviews app and verify that this request is denied with a 403 HTTP response code.kubectl --context ${REMOTE_CONTEXT1} -n bookinfo debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=reviews -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -vik www.google.com
Example output:
* Connected to www.google.com (240.240.59.19) port 80 (#0) > GET / HTTP/1.1 > Host: www.google.com > User-Agent: curl/8.1.1-DEV > Accept: */* > < HTTP/1.1 403 Forbidden < content-length: 19 < content-type: text/plain < date: Wed, 24 May 2023 20:58:55 GMT < server: envoy < x-envoy-upstream-service-time: 13 < { [19 bytes data] HTTP/1.1 403 Forbidden * Connection #0 to host www.google.com left intact content-length: 19 content-type: text/plain date: Wed, 24 May 2023 20:58:55 GMT server: envoy x-envoy-upstream-service-time: 13 RBAC: access denied%
Send another request to
www.google.com
from the product page app and verify that this request is accepted.kubectl --context ${REMOTE_CONTEXT1} -n bookinfo debug -i pods/$(kubectl get pod --context ${REMOTE_CONTEXT1} -l app=productpage -A -o jsonpath='{.items[0].metadata.name}') --image=curlimages/curl -- curl -vik www.google.com
Example output:
* Connected to www.google.com (240.240.59.19) port 80 (#0) > GET / HTTP/1.1 > Host: www.google.com > User-Agent: curl/8.1.1-DEV > Accept: */* > < HTTP/1.1 200 OK < date: Wed, 24 May 2023 21:01:44 GMT < expires: -1 < cache-control: private, max-age=0 < content-type: text/html; charset=ISO-8859-1 HTTP/1.1 200 OK
Cleanup
You can optionally remove the resources that you set up as part of this guide.
kubectl delete externalservice google -n global --context $REMOTE_CONTEXT1
kubectl delete virtualgateway egress-vg -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
kubectl delete gatewaylifecyclemanager istio-egressgateway -n gloo-mesh --context $MGMT_CONTEXT
kubectl delete accesspolicy egress-access-policy -n gloo-mesh-gateways --context $REMOTE_CONTEXT1
kubectl delete ns global --context $REMOTE_CONTEXT1