Hybrid Gateway
Hybrid Gateways allow users to define multiple HTTP or TCP Gateways for a single Gateway with distinct matching criteria.
Hybrid gateways expand the functionality of HTTP and TCP gateways by exposing multiple gateways on the same port and letting you use request properties to choose which gateway the request routes to.
Selection is done based on Matcher
fields, which map to a subset of Envoy FilterChainMatch
fields.
Only accept requests from a particular CIDR range
Hybrid Gateways allow us to treat traffic from particular IPs differently. One case where this might come in handy is if a set of clients are at different stages of migrating to TLS >=1.2 support, and therefore we want to enforce different TLS requirements depending on the client. If the clients originate from the same domain, it may be necessary to dynamically route traffic to the appropriate Gateway based on source IP.
In this example, we will allow requests only from a particular CIDR range to reach an upstream, while short-circuiting requests from all other IPs by using a direct response action.
Before you begin: Complete the Hello World guide demo setup.
To start we will add a second VirtualService that also matches all requests and has a directResponseAction:
kubectl apply -n gloo-system -f - <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: 'client-ip-reject'
namespace: 'gloo-system'
spec:
virtualHost:
domains:
- '*'
routes:
- matchers:
- prefix: /
directResponseAction:
status: 403
body: "client ip forbidden\n"
EOF
Next let’s update the existing gateway-proxy
Gateway CR, replacing the default httpGateway
with a hybridGateway
as follows:
kubectl edit -n gloo-system gateway gateway-proxy
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata: # collapsed for brevity
spec:
bindAddress: '::'
bindPort: 8080
hybridGateway:
matchedGateways:
- httpGateway:
virtualServices:
- name: default
namespace: gloo-system
matcher:
sourcePrefixRanges:
- addressPrefix: 0.0.0.0
prefixLen: 1
- httpGateway:
virtualServices:
- name: client-ip-reject
namespace: gloo-system
matcher: {}
proxyNames:
- gateway-proxy
useProxyProto: false
status: # collapsed for brevity
The range of 0.0.0.0/1 provides a high chance of matching the client’s IP without knowing the specific IP. If you know more about the client’s IP, you can specify a different, narrower range.
Make a request to the proxy, which returns a 200
response because the client IP address matches to the 0.0.0.0/1 range:
$ curl "$(glooctl proxy url)/all-pets"
[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]
Note that a request to an endpoint that is not matched by the default
VirtualService returns a 404
response, and the request does not hit the client-ip-reject
VirtualService:
$ curl -i "$(glooctl proxy url)/foo"
HTTP/1.1 404 Not Found
date: Tue, 07 Dec 2021 17:48:49 GMT
server: envoy
content-length: 0
This is because the Matcher
s in the HybridGateway
determine which MatchedGateway
a request will be routed to, regardless of what routes that gateway has.
Route requests from non-matching IPs to a catchall gateway
Next, update the matcher to use a specific IP range that our client’s IP is not a member of. Requests from this client IP will now skip this matcher, and will instead match to a catchall gateway that is configured to respond with 403
.
kubectl edit -n gloo-system gateway gateway-proxy
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata: # collapsed for brevity
spec:
bindAddress: '::'
bindPort: 8080
hybridGateway:
matchedGateways:
- httpGateway:
virtualServices:
- name: default
namespace: gloo-system
matcher:
sourcePrefixRanges:
- addressPrefix: 1.2.3.4
prefixLen: 32
- httpGateway:
virtualServices:
- name: client-ip-reject
namespace: gloo-system
matcher: {}
proxyNames:
- gateway-proxy
useProxyProto: false
status: # collapsed for brevity
The Proxy will update accordingly.
Make a request to the proxy, which now returns a 403
response for any endpoint:
$ curl "$(glooctl proxy url)/all-pets"
client ip forbidden
$ curl "$(glooctl proxy url)/foo"
client ip forbidden
This is expected since the IP of our client is not 1.2.3.4
.
Pass TLS traffic through for deprecated cipher suites
This feature requires a Gloo Gateway Enterprise license, and is available in version 1.14.3 or later.
By default, Gloo Gateway only supports cipher suites that are available in Envoy, which uses BoringSSL for TLS. If you provide services to clients that do not yet have support for the ciphers that are supported by BoringSSL, you can configure Gloo Gateway to pass TLS traffic through to your upstream directly. Because TLS traffic is not terminated at the gateway, the upstream service must be capable of terminating and unencrypting the incoming TLS connection.
In this guide, you deploy a sample NGINX server and configure the server for HTTPS traffic. You use the server to try out the TLS passthrough feature for deprecated cipher suites.
This guide assumes that you want to configure TLS passthrough in the gateway resource directly. If you have a hybrid gateway and you use the hybrid gateway delegation feature, follow the steps in Pass TLS traffic through for depcrecated cipher suites with hybrid gateway delegation.
-
Create a root certificate for the
example.com
domain. You use this certificate to sign the certificate for your NGINX service later.mkdir example_certs openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example_certs/example.com.key -out example_certs/example.com.crt
-
Create a server certificate and private key for the
nginx.example.com
domain.openssl req -out example_certs/nginx.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs/nginx.example.com.key -subj "/CN=nginx.example.com/O=some organization" openssl x509 -req -sha256 -days 365 -CA example_certs/example.com.crt -CAkey example_certs/example.com.key -set_serial 0 -in example_certs/nginx.example.com.csr -out example_certs/nginx.example.com.crt
-
Create a secret that stores the certificate and key for the NGINX server.
kubectl create secret tls nginx-server-certs \ --key example_certs/nginx.example.com.key \ --cert example_certs/nginx.example.com.crt
-
Prepare your NGINX configuration. The following example configures NGINX for HTTPS traffic with the certificate that you created earlier.
cat <<\EOF > ./nginx.conf events { } http { log_format main '$remote_addr - $remote_user [$time_local] $status ' '"$request" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; error_log /var/log/nginx/error.log; server { listen 443 ssl; root /usr/share/nginx/html; index index.html; server_name nginx.example.com; ssl_certificate /etc/nginx-server-certs/tls.crt; ssl_certificate_key /etc/nginx-server-certs/tls.key; } } EOF
-
Store the NGINX configuration in a configmap.
kubectl create configmap nginx-configmap --from-file=nginx.conf=./nginx.conf
-
Deploy the NGINX server.
kubectl apply -f- <<EOF apiVersion: v1 kind: Service metadata: name: my-nginx labels: run: my-nginx spec: ports: - port: 443 protocol: TCP selector: run: my-nginx EOF
kubectl apply -f- <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 1 template: metadata: labels: run: my-nginx sidecar.istio.io/inject: "true" spec: containers: - name: my-nginx image: nginx ports: - containerPort: 443 volumeMounts: - name: nginx-config mountPath: /etc/nginx readOnly: true - name: nginx-server-certs mountPath: /etc/nginx-server-certs readOnly: true volumes: - name: nginx-config configMap: name: nginx-configmap - name: nginx-server-certs secret: secretName: nginx-server-certs EOF
-
Verify that an upstream was created for your nginx server.
kubectl get upstream default-my-nginx-443 -n gloo-system
-
Create a gateway resource and specify the deprecated cipher suites for which you want to pass TLS traffic through to your upstream in the
passthroughCipherSuites
field of yourtcpGateway
. You can optionally log the ciphers that were sent by the clients by adding thespec.options.accessLoggingService
section to your gateway configuration as shown in the following example.kubectl apply -f- <<EOF apiVersion: gateway.solo.io/v1 kind: Gateway metadata: name: gateway-proxy-ssl namespace: gloo-system spec: bindAddress: '::' bindPort: 8443 options: accessLoggingService: accessLog: - fileSink: path: /dev/shm/access_log.txt stringFormat: > [%START_TIME%] %REQUESTED_SERVER_NAME% %FILTER_STATE(io.solo.cipher_detection_info)% hybridGateway: matchedGateways: - tcpGateway: options: tcpProxySettings: accessLogFlushInterval: 2s tcpHosts: - destination: single: upstream: name: default-my-nginx-443 namespace: gloo-system name: one matcher: passthroughCipherSuites: - ECDHE-RSA-AES256-SHA384 - ECDHE-RSA-AES128-SHA256 - AES256-SHA256 - AES128-SHA256 sslConfig: sniDomains: - nginx.example.com proxyNames: - gateway-proxy useProxyProto: false EOF
-
Get the IP address of the gateway.
export GATEWAY_IP=$(kubectl get svc -n gloo-system gateway-proxy -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
-
Send a request to the gateway. In your output, verify that you see the TLS handshake between the client and the gateway and that the cipher suite that you sent as part of the request is accepted. In addition, verify that you get back a 200 HTTP response code from the NGINX server and that you see the NGINX server certificate in your CLI output.
curl -vi --resolve nginx.example.com:443:${GATEWAY_IP} --cacert example_certs/nginx.example.com.crt --cipher "AES256-SHA256" "https://nginx.example.com:443"
Example output:
* Added nginx.example.com:443:34.134.215.40 to DNS cache * Hostname nginx.example.com was found in DNS cache * Trying 34.134.215.40:443... * Connected to nginx.example.com (34.134.215.40) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: AES256-SHA256 * successfully set certificate verify locations: * CAfile: example_certs/nginx.example.com.crt * CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / AES256-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=nginx.example.com; O=some organization * start date: May 22 18:46:27 2023 GMT * expire date: May 21 18:46:27 2024 GMT * common name: nginx.example.com (matched) * issuer: O=example Inc.; CN=example.com * SSL certificate verify ok. > GET / HTTP/1.1 > Host: nginx.example.com > User-Agent: curl/7.77.0 > Accept: */* * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK HTTP/1.1 200 OK
-
Send another request to the gateway. This time, you provide a cipher that is not listed in the deprecated cipher suites. Note that the TLS connection fails as no supported cipher can be found in the request.
curl -vi --resolve nginx.example.com:443:${GATEWAY_IP} --cacert example_certs/nginx.example.com.crt --cipher "AES128" "https://nginx.example.com:443"
Example output:
* Added nginx.example.com:443:34.134.215.40 to DNS cache * Hostname nginx.example.com was found in DNS cache * Trying 34.134.215.40:443... * Connected to nginx.example.com (34.134.215.40) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: AES128 * successfully set certificate verify locations: * CAfile: example_certs/nginx.example.com.crt * CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * LibreSSL SSL_connect: Connection reset by peer in connection to nginx.example.com:443 * Closing connection 0 curl: (35) LibreSSL SSL_connect: Connection reset by peer in connection to nginx.example.com:443
Hybrid Gateway Delegation
With Hybrid Gateways, you can define multiple HTTP and TCP Gateways, each with distinct matching criteria, on a single Gateway CR.
However, condensing all listener and routing configuration onto a single object can be cumbersome when dealing with a large number of matching and routing criteria.
Similar to how Gloo Gateway provides delegation between Virtual Services and Route Tables, Hybrid Gateways can be assembled from separate resources. The root Gateway resource selects HttpGateways and assembles the Hybrid Gateway, as though it were defined in a single resource.
Only accept requests from a particular CIDR range
We will use Hybrid Gateway delegation to achieve the same functionality that we demonstrated earlier in this guide.
-
Confirm that a Virtual Service exists which matches all requests and has a Direct Response Action.
kubectl get -n gloo-system vs client-ip-reject
-
Create a MatchableHttpGateway to define the HTTP Gateway.
kubectl apply -n gloo-system -f - <<EOF apiVersion: gateway.solo.io/v1 kind: MatchableHttpGateway metadata: name: client-ip-reject-gateway namespace: gloo-system spec: httpGateway: virtualServices: - name: client-ip-reject namespace: gloo-system matcher: {} EOF
-
Confirm the MatchableHttpGateway was created.
kubectl get -n gloo-system hgw client-ip-reject-gateway
-
Modify the Gateway CR to reference this MatchableHttpGateway.
kubectl edit -n gloo-system gateway gateway-proxy
apiVersion: gateway.solo.io/v1 kind: Gateway metadata: # collapsed for brevity spec: bindAddress: '::' bindPort: 8080 hybridGateway: delegatedHttpGateways: ref: name: client-ip-reject-gateway namespace: gloo-system proxyNames: - gateway-proxy useProxyProto: false status: # collapsed for brevity
-
Confirm expected routing behavior. We now have a Gateway which has matching and routing behavior defined in the
MatchableHttpGateway
. At this point, all requests (an empty matcher is treated as a match-all) are expected to be matched and delegated to theclient-ip-reject
Virtual Service.$ curl "$(glooctl proxy url)/all-pets" client ip forbidden
Although we demonstrate gateway delegation using reference selection in this guide, label selection is also supported.
Pass TLS traffic through for deprecated ciphers suites with hybrid gateway delegation
This feature requires a Gloo Gateway Enterprise license, and is available in version 1.14.3 or later.
By default, Gloo Gateway supports cipher suites that are available in Envoy. If you have an upstream service that can accept only cipher suites that are deprecated or not available in Envoy, you can configure Gloo Gateway to pass TLS traffic through to your upstream directly. Because TLS traffic is not terminated at the gateway, the upstream service must be capable of terminating and unencrypting the incoming TLS connection.
In this guide, you deploy a sample NGINX server and configure the server for HTTPS traffic. You use the server to try out the TLS passthrough feature for deprecated cipher suites.
This guide assumes that you want to use hybrid gateway delegation to configure TLS passthrough for your upstream. If you want to put this configuration in your gateway directly, follow the steps in Pass TLS traffic through for depcrecated cipher suites.
-
Follow steps 1-7 in Pass TLS traffic through for deprecated cipher suites to deploy the NGINX server and set it up for HTTPS traffic.
-
Create a
MatchableTCPGateway
kubectl apply -f- <<EOF apiVersion: gateway.solo.io/v1 kind: MatchableTcpGateway metadata: name: nginx-tpc-gateway namespace: gloo-system labels: protocol: tcp tls: passthrough spec: tcpGateway: options: tcpProxySettings: accessLogFlushInterval: 2s tcpHosts: - destination: single: upstream: name: default-my-nginx-443 namespace: gloo-system name: one matcher: passthroughCipherSuites: - ECDHE-RSA-AES256-SHA384 - ECDHE-RSA-AES128-SHA256 - AES256-SHA256 - AES128-SHA256 sslConfig: sniDomains: - nginx.example.com EOF
-
Create a gateway resource. In the
hybridGateway
section, add adelegatedTcpGateways
section and make sure that the protocol is set totcp
and the TLS mode topassthrough
. You can optionally log the ciphers that were sent by the clients by adding thespec.options.accessLoggingService
section to your gateway configuration as shown in the following example.kubectl apply -f- <<EOF apiVersion: gateway.solo.io/v1 kind: Gateway metadata: name: gateway-proxy-ssl namespace: gloo-system spec: bindAddress: '::' bindPort: 8443 options: accessLoggingService: accessLog: - fileSink: path: /dev/shm/access_log.txt stringFormat: > [%START_TIME%] %REQUESTED_SERVER_NAME% %FILTER_STATE(io.solo.cipher_detection_info)% hybridGateway: delegatedTcpGateways: selector: labels: protocol: tcp tls: passthrough proxyNames: - gateway-proxy useProxyProto: false EOF
-
Get the IP address of the gateway.
export GATEWAY_IP=$(kubectl get svc -n gloo-system gateway-proxy -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
-
Send a request to the gateway. In your output, verify that you see the TLS handshake between the client and the gateway and that the cipher suite that you sent as part of the request is accepted. In addition, verify that you get back a 200 HTTP response code from the NGINX server and that you see the NGINX server certificate in your CLI output.
curl -vi --resolve nginx.example.com:443:${GATEWAY_IP} --cacert example_certs/nginx.example.com.crt --cipher "AES256-SHA256" "https://nginx.example.com:443"
Example output:
* Added nginx.example.com:443:34.134.215.40 to DNS cache * Hostname nginx.example.com was found in DNS cache * Trying 34.134.215.40:443... * Connected to nginx.example.com (34.134.215.40) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: AES256-SHA256 * successfully set certificate verify locations: * CAfile: example_certs/nginx.example.com.crt * CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / AES256-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=nginx.example.com; O=some organization * start date: May 22 18:46:27 2023 GMT * expire date: May 21 18:46:27 2024 GMT * common name: nginx.example.com (matched) * issuer: O=example Inc.; CN=example.com * SSL certificate verify ok. > GET / HTTP/1.1 > Host: nginx.example.com > User-Agent: curl/7.77.0 > Accept: */* * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK HTTP/1.1 200 OK
-
Send another request to the gateway. This time, you provide a cipher that is not listed in the deprecated cipher suites. Note that the TLS connection fails as no supported cipher can be found in the request.
curl -vi --resolve nginx.example.com:443:${GATEWAY_IP} --cacert example_certs/nginx.example.com.crt --cipher "AES128" "https://nginx.example.com:443"
Example output:
* Added nginx.example.com:443:34.134.215.40 to DNS cache * Hostname nginx.example.com was found in DNS cache * Trying 34.134.215.40:443... * Connected to nginx.example.com (34.134.215.40) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: AES128 * successfully set certificate verify locations: * CAfile: example_certs/nginx.example.com.crt * CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * LibreSSL SSL_connect: Connection reset by peer in connection to nginx.example.com:443 * Closing connection 0 curl: (35) LibreSSL SSL_connect: Connection reset by peer in connection to nginx.example.com:443
How are SSL Configurations managed in Hybrid Gateways?
Before Hybrid Gateways were introduced, SSL configuration was exclusively defined on Virtual Services. This enabled the teams owning those services to define the SSL configuration required to establish connections.
With Hybrid Gateways, SSL configuration can also be defined in the matcher on the Gateway.
To support the legacy model, the SSL configuration defined on the Gateway acts as the default, and SSL configurations defined on the Virtual Services override that default. The presence of SSL configuration on the matcher determines whether a given matched Gateway will have any SSL configuration. Therefore one can define empty SSL configuration on Gateway matchers in order to exclusively use SSL configuration from Virtual Services.