You can follow this guide to customize settings for an advanced Gloo Network installation. To learn more about the benefits and architecture, see About.

About the Solo distribution of Cilium

Use Gloo Network for Cilium to provide connectivity, security, and observability for containerized workloads with a Cilium-based container network interface (CNI) plug-in that leverages the Linux kernel technology eBPF.

The Solo distribution of Cilium is a hardened Cilium enterprise image, which maintains support for security patches to address Common Vulnerabilities and Exposures (CVEs) and other security fixes.

Keep in mind that Gloo Network offers security patching support only with Solo distributions of Cilium versions, not community Cilium versions. Solo distributions of Cilium versions support the same patch versions as community Cilium. You can review community Cilium patch versions in the Cilium release documentation. To get the backported Cilium support, you must run the latest Gloo Network patch version.

To download the Solo distribution of a Cilium image, you must be a registered user and be able to log in to the Solo Support Center. Open the Cilium images built by Solo.io support article. When prompted, log in to the Support Center with your Solo account credentials.

The following versions of Gloo Network are supported with the compatible Solo versions of Cilium. Later versions of the open source project that are released after Gloo Network might also work, but are not tested as part of the Gloo Network release.

Gloo NetworkRelease dateSupported Solo distributions of Cilium versions tested by Solo
2.509 Jan 2024Cilium 1.12 - 1.14 on Kubernetes 1.22 - 1.28

Before you begin

  1. Install the following command-line (CLI) tools.

    • kubectl, the Kubernetes command line tool. Download the kubectl version that is within one minor version of the Kubernetes clusters you plan to use.
    • meshctl, the Solo command line tool.
        curl -sL https://run.solo.io/meshctl/install | GLOO_MESH_VERSION=v2.5.6 sh -
export PATH=$HOME/.gloo-mesh/bin:$PATH
    • helm, the Kubernetes package manager.
    • cilium, the Cilium command line tool.

  2. Create environment variables for the following details.

    • GLOO_NETWORK_LICENSE_KEY: Set your Gloo Network for Cilium license key as an environment variable. If you do not have one, contact an account representative.
    • GLOO_VERSION: Set the Gloo Network version. This example uses the latest version. Append -fips for a FIPS-compliant image. Do not include v before the version number.
    • SOLO_CILIUM_REPO: A repo key for the Solo distribution of Cilium that you can get by logging in to the Support Center and reviewing the Cilium images built by Solo.io support article.
    • CILIUM_VERSION: The Cilium version that you want to install. This example uses the latest version.
      export GLOO_NETWORK_LICENSE_KEY=<license_key>
export GLOO_VERSION=2.5.6
export SOLO_CILIUM_REPO=<cilium_repo_key>
export CILIUM_VERSION=1.14.2

  3. Optional: If you plan to run Istio with sidecar injection and the Cilium CNI in tunneling mode (VXLAN or GENEVE) on an Amazon EKS cluster, see Considerations for running Cilium and Istio on EKS.

Install the Solo distribution of the Cilium CNI

  1. Create or use Kubernetes clusters that meet the Cilium requirements. For example, to try out the Cilium CNI in Google Kubernetes Engine (GKE) clusters, your clusters must be created with specific node taints.

    1. Open the Cilium documentation and find the cloud provider that you want to use to create your clusters.

    2. Follow the steps of your cloud provider to create one or more clusters that meet the Cilium requirements.

      • The instructions in the Cilium documentation might create a cluster with insufficient CPU and memory resources for Gloo Network. Make sure that you use a machine type with at least 2vCPU and 8GB of memory.
      • The cluster name must be alphanumeric with no special characters except a hyphen (-), lowercase, and begin with a letter (not a number).
      • Multicluster setups only: For a multicluster setup, you need at least two clusters. One cluster is set up as the Gloo management plane where the management components are installed. The other cluster is registered as your data plane and runs your Kubernetes workloads and Istio service mesh. You can optionally add more workload clusters to your setup. The instructions in this guide assume one management cluster and two workload clusters.

      Example to create a cluster in GKE:

        export NAME="$(whoami)-$RANDOM"                                                        
gcloud container clusters create "${NAME}" \
--node-taints node.cilium.io/agent-not-ready=true:NoExecute \
--zone us-west2-a \
--machine-type e2-standard-2
gcloud container clusters get-credentials "${NAME}" --zone us-west2-a

  2. Add and update the Cilium Helm repo.

      helm repo add cilium https://helm.cilium.io/
helm repo update

  3. Install the CNI by using a Solo distribution of Cilium in your cluster. Be sure to include the following settings for compatability with Gloo Network, and the necessary settings for your infrastructure provider.

    Generic example (add cloud provider-specific values in --set flags below):

      helm install cilium cilium/cilium \
--namespace kube-system \
--version ${CILIUM_VERSION} \
--set customCalls.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enabled="{dns:destinationContext=pod;sourceContext=pod,drop:destinationContext=pod;sourceContext=pod,tcp:destinationContext=pod;sourceContext=pod,flow:destinationContext=pod;sourceContext=pod,port-distribution:destinationContext=pod;sourceContext=pod}" \
--set image.repository=${SOLO_CILIUM_REPO}/cilium \
--set image.useDigest=false \
--set operator.image.repository=${SOLO_CILIUM_REPO}/operator \
--set operator.image.useDigest=false \
--set operator.prometheus.enabled=true \
--set prometheus.enabled=true \
# ADD CLOUD PROVIDER-SPECIFIC HELM VALUES HERE

    Example for GKE:

      NATIVE_CIDR="$(gcloud container clusters describe "${NAME}" --zone "${ZONE}" --format 'value(clusterIpv4Cidr)')"
echo $NATIVE_CIDR

helm install cilium cilium/cilium \
--namespace kube-system \
--version ${CILIUM_VERSION} \
--set customCalls.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enabled="{dns:destinationContext=pod;sourceContext=pod,drop:destinationContext=pod;sourceContext=pod,tcp:destinationContext=pod;sourceContext=pod,flow:destinationContext=pod;sourceContext=pod,port-distribution:destinationContext=pod;sourceContext=pod}" \
--set image.repository=${SOLO_CILIUM_REPO}/cilium \
--set image.useDigest=false \
--set operator.image.repository=${SOLO_CILIUM_REPO}/operator \
--set operator.image.useDigest=false \
--set operator.prometheus.enabled=true \
--set prometheus.enabled=true \
--set nodeinit.enabled=true \
--set nodeinit.reconfigureKubelet=true \
--set nodeinit.removeCbrBridge=true \
--set cni.binPath=/home/kubernetes/bin \
--set gke.enabled=true \
--set ipam.mode=kubernetes \
--set ipv4NativeRoutingCIDR=${NATIVE_CIDR}

    Generic example (add any cloud provider-specific values):

      helm upgrade -i -n kube-system cilium cilium/cilium --version ${CILIUM_VERSION} -f - <<EOF
customCalls:
  enabled: true
hubble:
  enabled: true
  metrics:
    enabled:
    - dns:destinationContext=pod;sourceContext=pod
    - drop:destinationContext=pod;sourceContext=pod
    - tcp:destinationContext=pod;sourceContext=pod
    - flow:destinationContext=pod;sourceContext=pod
    - port-distribution:destinationContext=pod;sourceContext=pod
image:
  repository: ${SOLO_CILIUM_REPO}/cilium
  useDigest: false
operator:
  replicas: 1
  prometheus:
    enabled: true
  image:
    repository: ${SOLO_CILIUM_REPO}/operator
    useDigest: false
prometheus:
  enabled: true
# ADD CLOUD PROVIDER-SPECIFIC HELM VALUES
EOF

    Example for GKE:

      NATIVE_CIDR="$(gcloud container clusters describe "${NAME}" --zone "${ZONE}" --format 'value(clusterIpv4Cidr)')"
echo $NATIVE_CIDR

helm upgrade -i -n kube-system cilium cilium/cilium --version ${CILIUM_VERSION} -f - <<EOF
customCalls:
  enabled: true
hubble:
  enabled: true
  metrics:
    enabled:
    - dns:destinationContext=pod;sourceContext=pod
    - drop:destinationContext=pod;sourceContext=pod
    - tcp:destinationContext=pod;sourceContext=pod
    - flow:destinationContext=pod;sourceContext=pod
    - port-distribution:destinationContext=pod;sourceContext=pod
image:
  repository: ${SOLO_CILIUM_REPO}/cilium
  useDigest: false
operator:
  replicas: 1
  prometheus:
    enabled: true
  image:
    repository: ${SOLO_CILIUM_REPO}/operator
    useDigest: false
prometheus:
  enabled: true
nodeinit:
  enabled: true
  reconfigureKubelet: true
  removeCbrBridge: true
cni:
  binPath: /home/kubernetes/bin
gke:
  enabled: true
ipam:
  mode: kubernetes
ipv4NativeRoutingCIDR: ${NATIVE_CIDR}
EOF

    Example output:

      NAME: cilium
LAST DEPLOYED: Fri Sep 16 10:31:52 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.14.2.

For any further help, visit https://docs.cilium.io/en/v1.12/gettinghelp

  4. Verify that the Cilium CNI is successfully installed. Because the Cilium agent is deployed as a daemon set, the number of Cilium and Cilium node init pods equals the number of nodes in your cluster.

      kubectl get pods -n kube-system | grep cilium

    Example output:

      cilium-gbqgq                                                  1/1     Running             0          48s
cilium-j9n5x                                                  1/1     Running             0          48s
cilium-node-init-c7rxb                                        1/1     Running             0          48s
cilium-node-init-pnblb                                        1/1     Running             0          48s
cilium-node-init-wdtjm                                        1/1     Running             0          48s
cilium-operator-69dd4567b5-2gjgg                              1/1     Running             0          47s
cilium-operator-69dd4567b5-ww6wp                              1/1     Running             0          47s
cilium-smp9c                                                  1/1     Running             0          48s

  5. Check the status of the Cilium installation.

      cilium status --wait

    Example output:

      
____/¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet              cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:            cilium-operator    Running: 2
                       cilium             Running: 4
Cluster Pods:          3/3 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             ${SOLO_CILIUM_REPO}/cilium:v1.14.2: 4
                       cilium-operator    ${SOLO_CILIUM_REPO}/operator-generic:v1.14.2: 2

  6. Multicluster setups only: Repeat steps 3 - 5 to install the CNI in each cluster that you want to use in your Gloo Network environment.

Install Gloo Network for Cilium

Deploy the Gloo Network components into one cluster or across a multicluster environment.

Single cluster

Install the Gloo Network for Cilium components in your cluster and verify that Gloo Network can discover the Cilium CNI.

  1. Save the name of your cluster as an environment variable.

      export CLUSTER_NAME=<cluster_name>

  2. Add and update the Helm repository for Gloo.

      helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

  3. Install the Gloo CRDs.

      helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
   --namespace=gloo-mesh \
   --create-namespace \
   --version=$GLOO_VERSION \
   --set installEnterpriseCrds=false

  4. Prepare a Helm values file to provide your customizations. To get started, you can use the minimum settings in the following profile as a basis. These settings enable all components that are required for a single-cluster Gloo Network installation.

      curl https://storage.googleapis.com/gloo-platform/helm-profiles/$GLOO_VERSION/gloo-core-single-cluster.yaml > gloo-network-single.yaml
open gloo-network-single.yaml

  5. Decide how you want to secure the relay connection between the Gloo management server and agent. In test and POC environments, you can use Gloo Gateway self-signed certificates to secure the connection. If you plan to use Gloo Gateway in production, it is recommended to bring your own certificates instead. For more information, see Setup options.

    Use Gloo Gateway to create a self-signed server TLS certificate that the Gloo management server presents when the Gloo agent connects. This setup is not recommended for production, but can be used for testing purposes. For more information, see Self-signed server TLS certificate.

    In your Helm values file, add the following values.

      
glooMgmtServer:
  serviceType: ClusterIP
  registerCluster: true
  enabled: true
  extraEnvs: 
    RELAY_TOKEN: 
      value: "My token"
    RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION: 
      value: "true" 
glooAgent:
  enabled: true
  relay:
    serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900
  extraEnvs:
    RELAY_TOKEN: 
      value: "My token"
    RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATION: 
      value: "true"
    Helm valueDescription
    glooMgmtServer.extraEnvs.RELAY_TOKENSpecify the relay token that the Gloo management server and agent use to establish initial trust. When you install Gloo Gateway and set RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION to true, the connection between the Gloo management server and agent is automatically secured by using simple, server-side TLS. In a simple TLS setup, only the management server presents a certificate to authenticate its identity. The identity of the agent is not verified. To ensure that only trusted agents connect to the management server, the relay identity token is used. The relay identity token can be any string value and is stored in the relay-identity-token-secret Kubernetes secret. You must set the same value in glooAgent.extraEnvs.RELAY_TOKEN.value to allow the Gloo agent to connect to the Gloo management server.
    glooMgmtServer.extraEnvs.
    RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION    		Set this value to true to not require a client TLS certificate from the Gloo agent to prove the agent’s identity and establish the connection with the management server. This setting is required when you want to use simple TLS to secure the connection between the Gloo management server and agent.
    glooAgent.extraEnvs.RELAY_TOKENUse the same value that you set in glooMgmtServer.extraEnvs.RELAY_TOKEN.
    glooAgent.extraEnvs.
    RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATION    		Set to true to skip validating the server TLS certificate that the Gloo management server presents. This setting is required to configure the relay connection for TLS.

    Use your preferred PKI provider to create the server TLS certificate for the Gloo management server. For more information, see Bring your own server TLS certificate.

    1. Create your root CA credentials and derive the server TLS certificate and certificate chain information. To create the TLS certificate, you can use your preferred PKI provider or follow the steps in BYO server certificates to create a custom server TLS certificate by using OpenSSL. Store this information in the following secrets in the gloo-mesh namespace.
      • relay-server-tls-secret-custom: The key and TLS certificate for the Gloo management server, and root CA certificate information for the certificate chain. Note that you can use a different name, but you cannot use relay-server-tls-secret as this name is reserved by the Gloo management server when creating self-signed root CAs and server TLS certificates.
      • telemetry-root-secret: The root CA certificate for the certificate chain.
    2. In your Helm values file, add the following values.
        
glooMgmtServer:
  serviceType: ClusterIP
  registerCluster: true
  enabled: true
  relay: 
    tlsSecret:
      name: relay-server-tls-secret-custom
      namespace: gloo-mesh
  extraEnvs: 
    RELAY_TOKEN: 
      value: "My token"
    RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION: 
      value: "true" 
glooAgent:
  enabled: true
  relay:
    serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900
  extraEnvs:
    RELAY_TOKEN: 
      value: "My token"
    RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATION: 
      value: "true"
telemetryCollector:
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
      Helm valueDescription
      glooMgmtServer.relay.tlsSecret.nameThe name and namespace of the Kubernetes secret where you stored your custom server TLS certificate.
      glooMgmtServer.extraEnvs.RELAY_TOKENSpecify the relay token that the Gloo management server and agent use to establish initial trust. When you install Gloo Gateway and set RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION to true, the connection between the Gloo management server and agent is automatically secured by using simple, server-side TLS. In a simple TLS setup, only the management server presents a certificate to authenticate its identity. The identity of the agent is not verified. To ensure that only trusted agents connect to the management server, the relay identity token is used. The relay identity token can be any string value and is stored in the relay-identity-token-secret Kubernetes secret on the management cluster. You must set the same value in glooAgent.extraEnvs.RELAY_TOKEN.value to allow the Gloo agent to connect to the Gloo management server.
      glooMgmtServer.extraEnvs.
      RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION      		Set this value to true to not require a client TLS certificate from the Gloo agent to prove the agent’s identity and establish the connection with the management server.
      glooAgent.extraEnvs.
      RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATION      		Set to true to skip validating the server TLS certificate that the Gloo management server presents. This setting is required to configure the relay connection for TLS.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

    You can use Gloo to generate self-signed certificates and keys for the root and intermediate CA. These credentials are used to derive the server TLS certificate for the Gloo management server and client TLS certificate for the Gloo agent. Note that this setup is not recommended for production, but can be used for testing purposes. For more information, see Self-signed CAs with automatic client certificate rotation.

    In your Helm values file, add the following values. Note that mTLS is the default mode in Gloo Gateway and does not require any additional configuration.

      
glooMgmtServer:
  enabled: true
glooAgent: 
  enabled: true

    Use your preferred PKI provider to generate your root CA and intermediate CA credentials. You then store the intermediate CA credentials in your cluster so that you can leverage Gloo Gateway’s built-in capability to automatically issue and sign the client TLS certificate for the Gloo agent. For more information, see the Bring your own CAs with automatic client TLS certificate rotation.

    1. Create your root CA, intermediate CA credentials, server TLS certificate, and relay identity token, and store them as the following Kubernetes secrets in the gloo-mesh namespace:

      • relay-root-tls-secret that holds the root CA key, TLS certificate, and certificate chain
      • relay-tls-signing-secret that holds the intermediate CA key, TLS certificate, and certificate chain
      • relay-server-tls-secret that holds the server key, TLS certificate, and certificate chain for the Gloo management server
      • telemetry-root-secret that holds the certificate chain information for the telemetry collector agent
      • relay-identity-token-secret that holds the relay identity token value

      You can use your preferred PKI provider to create the TLS certificates or follow the steps in BYOserver certificate with managed client certificate to create the relay identity token and the TLS certificate by using OpenSSL.

    2. In your Helm values file, add the following values.

        glooMgmtServer: 
  enabled: true
  registerCluster: true
  serviceType: ClusterIP
  relay:
    disableCaCertGeneration: true
    signingTlsSecret:
      name: relay-tls-signing-secret
      namespace: gloo-mesh
    tlsSecret:
      name: relay-server-tls-secret
      namespace: gloo-mesh
    tokenSecret:
      key: token
      name: relay-identity-token-secret
      namespace: gloo-mesh
glooAgent: 
  enabled: true
  relay:
    authority: gloo-mesh-mgmt-server.gloo-mesh
    serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900
    clientTlsSecretRotationGracePeriodRatio: ""
    rootTlsSecret:
      name: relay-root-tls-secret
      namespace: gloo-mesh
    tokenSecret:
      key: token
      name: relay-identity-token-secret
      namespace: gloo-mesh
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
      Helm valueDescription
      glooMgmtServer.relay.disableCaCertGenerationDisable the generation of self-signed certificates to secure the relay connection between the Gloo management server and agent.
      glooMgmtServer.relay.signingTlsSecretAdd the name and namespace of the Kubernetes secret that holds the intermediate CA credentials that you created earlier.
      glooMgmtServer.relay.tlsSecretAdd the name and namespace of the Kubernetes secret that holds the server TLS certificate for the Gloo management server that you created earlier.
      glooMgmtServer.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you created earlier.
      glooAgent.relay.rootTlsSecretAdd the name and namespace of the Kubernetes secret that holds the root CA credentials that you created earlier.
      glooAgent.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you created earlier.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

    Use your preferred PKI provider to create the root CA, server TLS, and client TLS certificates. Then, store these certificates in the cluster so that the Gloo agent uses a client TLS certificate when establishing the first connection with the Gloo management server. No relay identity tokens are used. With this approach, you cannot use Gloo Gateway’s built-in client TLS certificate rotation capabilities. Instead, you must set up your own processes to monitor the expiration of your certificates and to rotate them before they expire. For more information, see Bring your own CAs and client TLS certificates.

    1. Create your root CA, server, and client TLS certificates, and store them as the following Kubernetes secrets in the gloo-mesh namespace:

      • relay-server-tls-secret that holdes the server TLS certificate for the Gloo management server
      • relay-client-tls-cert that holds the client TLS certificate for the Gloo agent
      • telemetry-root-secret that holds the certificate chain information for the telemetry collector agent

      You can use your preferred PKI provider to create the TLS certificates or follow the steps in Create TLS certificates with OpenSSL to create them by using OpenSSL.

    2. In your Helm values file for the management server, add the following values.

        
glooMgmtServer:
  enabled: true
  registerCluster: true
  relay:
    disableCa: true
    disableCaCertGeneration: true
    disableTokenGeneration: true
    tlsSecret:
      name: relay-server-tls-secret
      namespace: gloo-mesh
    tokenSecret:
      key: null
      name: null
      namespace: null
  serviceType: ClusterIP
glooAgent:
  enabled: true
  relay:
    clientTlsSecret:
      name: relay-client-tls-cert
      namespace: gloo-mesh
    serverAddress: gloo-mesh-mgmt-server.gloo-mesh:9900
    tokenSecret:
      key: null
      name: null
      namespace: null
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
      Helm valueDescription
      glooMgmtServer.relay.disableCaDisable the generation of self-signed root and intermediate CA certificates and the use of identity tokens to establish initial trust between the Gloo management server and agent.
      glooMgmtServer.relay.disableCaCertGenerationDisable the generation of self-signed certificates to secure the relay connection between the Gloo management server and agent.
      glooMgmtServer.relay.disableTokenGenerationDisable the generation of relay identity tokens.
      glooMgmtServer.relay.tlsSecretAdd the name and namespace of the Kubernetes secret that holds the server TLS certificate for the Gloo management server that you created earlier.
      glooMgmtServer.relay.tokenSecretSet all values to null to instruct the Gloo management server to not use identity tokens to establish initial trust with Gloo agents.
      glooAgent.relay.clientTlsSecretAdd the name and namespace of the Kubernetes secret that holds the client TLS certificate for the Gloo agent that you created earlier.
      glooAgent.tokenSecretSet all values to null to instruct the Gloo agent to not use identity tokens to establish initial trust with the Gloo management server.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

  6. Edit the file to provide your own details for settings that are recommended for production deployments, such as the following settings.

    FieldDecription
    glooMgmtServer.resources.limitsAdd resource limits for the gloo-mesh-mgmt-server pod, such as cpu: 1000m and memory: 1Gi.
    glooMgmtServer.safeMode
    glooMgmtServer.safeStartWindow    		Configure how you want the Gloo management server to handle translation after a Redis restart. For available options, see Redis safe mode options.
    glooMgmtServer.serviceOverrides.metadata.annotationsAdd annotations for the management server load balancer as needed, such as AWS-specific load balancer annotations. For more information, see Deployment and service overrides.
    glooUi.authSet up OIDC authorization for the Gloo UI. For more information, see UI authentication.
    prometheus.enabledDisable the default Prometheus instance as needed to provide your own. Otherwise, you can keep the default Prometheus server enabled, and deploy a production-level server to scrape metrics from the server. For more information on each option, see Best practices for collecting metrics in production.
    redisDisable the default Redis deployment and provide your own backing database as needed. For more information, see Backing databases.
    glooNetwork.enabledSet to true to install the Gloo Network-specific agent, which collects additional metrics when Cilium is installed.
    telemetryCollectorCustomization
    • Cilium logs: To collect Cilium pod logs and view them in the Gloo UI, set telemetryCollectorCustomization.pipelines.logs/cilium_flows.enabled to true. For more information, see Add Cilium flow logs.
    • Cilium insights: To generate insights that analyze your Cilium setup’s health, set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled to true. This pipeline uses the default filter/cilium processor that collects all Cilium and Hubble metrics. You can optionally customize the Cilium metrics collection to reduce the number of metrics that are collected.

  7. Use the customizations in your Helm values file and the following recommended Cilium settings to install the Gloo Network components in your cluster.

      helm upgrade -i gloo-platform gloo-platform/gloo-platform \
    -n gloo-mesh \
    --version $GLOO_VERSION \
    --values gloo-network-single.yaml \
    --set common.cluster=$CLUSTER_NAME \
    --set licensing.glooNetworkLicenseKey=$GLOO_NETWORK_LICENSE_KEY \
    --set telemetryCollectorCustomization.pipelines.logs/cilium_flows.enabled=true \
    --set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled=true

  8. Verify that your Gloo Network setup is correctly installed. If not, try debugging the relay connection. Note that this check might take a few seconds to verify that:

    • Your Gloo product license is valid and current.
    • The Gloo CRDs are installed at the correct version.
    • The management plane pods in the management cluster are running and healthy.
    • The Gloo agent is running and connected to the management server.
      meshctl check

    Example output:

      🟢 License status

INFO  gloo-network enterprise license expiration is 25 Aug 24 10:38 CDT
INFO  No GraphQL license module found for any product

🟢 CRD version check

🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-mgmt-server          | 1/1   | Healthy
gloo-mesh | gloo-mesh-redis                | 1/1   | Healthy
gloo-mesh | gloo-mesh-ui                   | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 3/3   | Healthy
gloo-mesh | prometheus-server              | 1/1   | Healthy

🟢 Mgmt server connectivity to workload agents

Cluster | Registered | Connected Pod                                   
test    | true       | gloo-mesh/gloo-mesh-mgmt-server-558cddbbd7-rf2hv

Connected Pod                                    | Clusters
gloo-mesh/gloo-mesh-mgmt-server-558cddbbd7-rf2hv | 1

Multicluster

In a multicluster setup, you deploy the Gloo management plane into a dedicated management cluster, and the Gloo data plane into one or more workload clusters.

Management plane

Deploy the Gloo management plane into a dedicated management cluster.

  1. Save the name and kubeconfig context for your management cluster in environment variables.

      export MGMT_CLUSTER=<management-cluster-name>
export MGMT_CONTEXT=<management-cluster-context>

  2. Add and update the Helm repository for Gloo.

      helm repo add gloo-platform https://storage.googleapis.com/gloo-platform/helm-charts
helm repo update

  3. Install the Gloo CRDs.

      helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
   --namespace=gloo-mesh \
   --create-namespace \
   --version=$GLOO_VERSION \
   --set installEnterpriseCrds=false \
   --kube-context $MGMT_CONTEXT
  4. Prepare a Helm values file to provide your customizations. To get started, you can use the minimum settings in the following profile as a basis. These settings enable all components that are required for a Gloo management plane installation.
      curl https://storage.googleapis.com/gloo-platform/helm-profiles/$GLOO_VERSION/gloo-core-mgmt.yaml > mgmt-plane.yaml
open mgmt-plane.yaml

  5. Decide how you want to secure the relay connection between the Gloo management server and agents. In test and POC environments, you can use self-signed certificates to secure the connection. If you plan to use Gloo Network in production, it is recommended to bring your own certificates instead. For more information, see Setup options.

    Use Gloo Network to create a self-signed server TLS certificate that the Gloo management server presents when workload cluster agents connect. This setup is not recommended for production, but can be used for testing purposes. For more information, see Self-signed server TLS certificate.

    In your Helm values file for the management server, add the following values.

      
glooMgmtServer:
  enabled: true
  extraEnvs:
    RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION:
      value: "true"  
    RELAY_TOKEN: 
      value: "My token"
    Helm valueDescription
    RELAY_TOKENSpecify the relay token that the Gloo management server and agent use to establish initial trust. When you install Gloo Network and set RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION to true, the connection between the Gloo management server and agent is automatically secured by using simple, server-side TLS. In a simple TLS setup, only the management server presents a certificate to authenticate its identity. The identity of the agent is not verified. To ensure that only trusted agents connect to the management server, the relay identity token is used. The relay identity token can be any string value and is stored in the relay-identity-token-secret Kubernetes secret on the management cluster. You must set the same value in glooAgent.extraEnvs.RELAY_TOKEN.value when installing Gloo Network in a workload cluster to allow Gloo agents to connect to the Gloo management server.
    RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATIONSet this value to true to not require a client TLS certificate from the Gloo agent to prove the agent’s identity and establish the connection with the management server. This setting is required when you want to use simple TLS to secure the connection between the Gloo management server and agent.

    Use your preferred PKI provider to create the server TLS certificate for the Gloo management server. For more information, see Bring your own server TLS certificate.

    1. Create your root CA credentials and derive a server TLS certificate. To create the TLS certificate, you can use your preferred PKI provider or follow the steps in BYO server certificate to create a custom server TLS certificate by using OpenSSL. Make sure that you have the following Kubernetes secrets in the management cluster:

      • relay-server-tls-secret-custom: The key and TLS certificate for the Gloo management server, and root CA certificate information for the certificate chain. Note that you can use a different name, but you cannot use relay-server-tls-secret as this name is reserved by the Gloo management server when creating self-signed root CAs and server TLS certificates.
      • gloo-telemetry-gateway-tls-secret-custom: The key and TLS certificate for the Gloo telemetry gateway, and root CA certificate information for the certificate chain. It is recommended to use the same credentials that you use for the Gloo management server. Note that you can use a different name, but you cannot use gloo-telemetry-gateway-tls-secret as this name is reserved by the Gloo management server when creating self-signed certificates.
      • telemetry-root-secret: The root CA certificate for the certificate chain.

    2. In your Helm values file for the management server, add the following values.

        
glooMgmtServer:
  enabled: true
  relay:
    tlsSecret:
      name: relay-server-tls-secret-custom
  extraEnvs:
    RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION:
      value: "true"  
    RELAY_TOKEN: 
      value: "My token"
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
telemetryGateway:
  enabled: true
  extraVolumes:
  - name: tls-keys
    secret:
      secretName:  gloo-telemetry-gateway-tls-secret-custom
      defaultMode: 420
  - name: telemetry-configmap
    configMap:
      name: gloo-telemetry-gateway-config
      items:
        - key: relay
          path: relay.yaml
telemetryGatewayCustomization:
  disableCertGeneration: true
      Helm valueDescription
      relay.tlsSecret.nameAdd the name of the Kubernetes secret with the custom server TLS secret that you created earlier.
      RELAY_TOKENSpecify the relay token that the Gloo management server and agent use to establish initial trust. When you install Gloo Network and set RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATION to true, the connection between the Gloo management server and agent is automatically secured by using simple, server-side TLS. In a simple TLS setup, only the management server presents a certificate to authenticate its identity. The identity of the agent is not verified. To ensure that only trusted agents connect to the management server, the relay identity token is used. The relay identity token can be any string value and is stored in the relay-identity-token-secret Kubernetes secret on the management cluster. You must set the same value in glooAgent.extraEnvs.RELAY_TOKEN.value when you install the Gloo agent to allow the Gloo agent to connect to the Gloo management server.
      RELAY_DISABLE_CLIENT_CERTIFICATE_AUTHENTICATIONSet this value to true to not require a client TLS certificate from the Gloo agent to prove the agent’s identity and establish the connection with the management server. This setting is required when you want to use simple TLS to secure the connection between the Gloo management server and agent.
      telemetryGateway.extraVolumesAdd the gloo-telemetry-gateway-tls-secret-custom Kubernetes secret that you created earlier to the tls-keys volume. Make sure that you also add the other volumes to your telemetry gateway configuration.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

    You can use Gloo to generate self-signed certificates and keys for the root and intermediate CA. These credentials are used to derive the server TLS certificate for the Gloo management server and client TLS certificate for the Gloo agent. Note that this setup is not recommended for production, but can be used for testing purposes. For more information, see Self-signed CAs with automatic client certificate rotation.

    In your Helm values file for the management server, add the following values. Note that mTLS is the default mode in Gloo Network and does not require any additional configuration on the management server.

      
glooMgmtServer:
  enabled: true

    Use your preferred PKI provider to generate your root CA and intermediate CA credentials. You then store the intermediate CA credentials in your cluster so that you can leverage Gloo Network’s built-in capability to automatically issue and sign client TLS certificates for Gloo agents. For more information, see the Bring your own CAs with automatic client TLS certificate rotation.

    1. Create your root CA, intermediate CA, server, and telemetry gateway credentials, as well as relay identity token, and store them as the following Kubernetes secrets in the gloo-mesh namespace on the management cluster:

      • relay-root-tls-secret that holds the root CA certificate
      • relay-tls-signing-secret that holds the intermediate CA credentials
      • relay-server-tls-secret that holds the server key, TLS certificate and certificate chain information for the Gloo management server
      • relay-identity-token-secret that holds the relay identity token value
      • telemetry-root-secret that holds the root CA certificate for the certificate chain
      • gloo-telemetry-gateway-tls-secret that holds the key, TLS certificate and certificate chain for the Gloo telemetry gateway

      You can use your preferred PKI provider to create the TLS certificates or follow the steps in BYO server certificate with managed client certificate to create these TLS credentials with OpenSSl.

    2. In your Helm values file for the management server, add the following values.

        
glooMgmtServer:
  enabled: true
  relay:
    disableCaCertGeneration: true
    signingTlsSecret:
      name: relay-tls-signing-secret
      namespace: gloo-mesh
    tlsSecret:
      name: relay-server-tls-secret
      namespace: gloo-mesh
    tokenSecret:
      key: token
      name: relay-identity-token-secret
      namespace: gloo-mesh
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
telemetryGateway:
  enabled: true
  extraVolumes:
  - name: tls-keys
    secret:
      secretName:  gloo-telemetry-gateway-tls-secret
      defaultMode: 420
  - name: telemetry-configmap
    configMap:
      name: gloo-telemetry-gateway-config
      items:
        - key: relay
          path: relay.yaml
telemetryGatewayCustomization:
  disableCertGeneration: true
      Helm valueDescription
      glooMgmtServer.relay.disableCaCertGenerationDisable the generation of self-signed certificates to secure the relay connection between the Gloo management server and agent.
      glooMgmtServer.relay.signingTlsSecretAdd the name and namespace of the Kubernetes secret that holds the intermediate CA credentials that you created earlier.
      glooMgmtServer.relay.tlsSecretAdd the name and namespace of the Kubernetes secret that holds the server TLS certificate for the Gloo management server that you created earlier.
      glooMgmtServer.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you created earlier.
      telemetryGateway.extraVolumesAdd the gloo-telemetry-gateway-tls-secret-custom Kubernetes secret that you created earlier to the tls-keys volume. Make sure that you also add the other volumes to your telemetry gateway configuration.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

    Use your preferred PKI provider to create the root CA, server TLS, client TLS, and telemetry gateway certificates. Then, store these certificates in the cluster so that the Gloo agent uses a client TLS certificate when establishing the first connection with the Gloo management server. No relay identity tokens are used. With this approach, you cannot use Gloo Network’s built-in client TLS certificate rotation capabilities. Instead, you must set up your own processes to monitor the expiration of your certificates and to rotate them before they expire. For more information, see Bring your own CAs and client TLS certificates.

    1. Create your root CA, server, client, and telemetry gateway TLS certificates. You can use your preferred PKI provider to do that or follow the steps in Create certificates with OpenSSL to create custom TLS certificates by using OpenSSL. Then, you store the following information in a Kubernetes secret in the gloo-mesh namespace on the management cluster.
      • relay-server-tls-secret that holds the server key, TLS certificate and certificate chain information for the Gloo management server
      • gloo-telemetry-gateway-tls-secret that holds the key, TLS certificate and certificate chain for the Gloo telemetry gateway
      • telemetry-root-secret that holds the root CA certificate for the certificate chain
    2. In your Helm values file for the management server, add the following values.
        
glooMgmtServer:
  enabled: true
  relay:
    disableCa: true
    disableCaCertGeneration: true
    disableTokenGeneration: true
    tlsSecret:
      name: relay-server-tls-secret
      namespace: gloo-mesh
    tokenSecret:
      key: null
      name: null
      namespace: null
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
telemetryGateway:
  enabled: true
  extraVolumes:
  - name: tls-keys
    secret:
      secretName:  gloo-telemetry-gateway-tls-secret
      defaultMode: 420
  - name: telemetry-configmap
    configMap:
      name: gloo-telemetry-gateway-config
      items:
        - key: relay
          path: relay.yaml
telemetryGatewayCustomization:
  disableCertGeneration: true
      Helm valueDescription
      relay.disableCaDisable the generation of self-signed root and intermediate CA certificates and the use of identity tokens to establish initial trust between the Gloo management server and agent.
      relay.disableCaCertGenerationDisable the generation of self-signed certificates to secure the relay connection between the Gloo management server and agent.
      relay.disableTokenGenerationDisable the generation of relay identity tokens.
      relay.tlsSecretAdd the name and namespace of the Kubernetes secret that holds the server TLS certificate for the Gloo management server that you created earlier.
      relay.tokenSecretSet all values to null to instruct the Gloo management server to not use identity tokens to establish initial trust with Gloo agents.
      telemetryGateway.extraVolumesAdd the gloo-telemetry-gateway-tls-secret Kubernetes secret that you created earlier to the tls-keys volume. Make sure that you also add the other volumes to your telemetry gateway configuration.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

  6. Edit the file to provide your own details for settings that are recommended for production deployments, such as the following settings.

    FieldDecription
    glooMgmtServer.resources.limitsAdd resource limits for the gloo-mesh-mgmt-server pod, such as cpu: 1000m and memory: 1Gi.
    glooMgmtServer.safeMode
    glooMgmtServer.safeStartWindow    		Configure how you want the Gloo management server to handle translation after a Redis restart. For available options, see Redis safe mode options.
    glooMgmtServer.serviceOverrides.metadata.annotationsAdd annotations for the management server load balancer as needed, such as AWS-specific load balancer annotations. For more information, see Deployment and service overrides.
    glooUi.authSet up OIDC authorization for the Gloo UI. For more information, see UI authentication.
    prometheus.enabledDisable the default Prometheus instance as needed to provide your own. Otherwise, you can keep the default Prometheus server enabled, and deploy a production-level server to scrape metrics from the server. For more information on each option, see Best practices for collecting metrics in production.
    redisDisable the default Redis deployment and provide your own backing database as needed. For more information, see Backing databases.
    glooNetwork.enabledSet to true to install the Gloo Network-specific agent, which collects additional metrics when Cilium is installed.
    telemetryCollectorCustomization
    • Cilium logs: To collect Cilium pod logs and view them in the Gloo UI, set telemetryCollectorCustomization.pipelines.logs/cilium_flows.enabled to true. For more information, see Add Cilium flow logs.
    • Cilium insights: To generate insights that analyze your Cilium setup’s health, set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled to true. This pipeline uses the default filter/cilium processor that collects all Cilium and Hubble metrics. You can optionally customize the Cilium metrics collection to reduce the number of metrics that are collected.

  7. Use the customizations in your Helm values file to install the Gloo management plane components in your management cluster.

      helm upgrade -i gloo-platform gloo-platform/gloo-platform \
    --kube-context $MGMT_CONTEXT \
    -n gloo-mesh \
    --version $GLOO_VERSION \
    --values mgmt-plane.yaml \
    --set common.cluster=$MGMT_CLUSTER \
    --set licensing.glooNetworkLicenseKey=$GLOO_NETWORK_LICENSE_KEY \
    --set glooNetwork.enabled=true \
    --set telemetryCollectorCustomization.pipelines.logs/cilium_flows.enabled=true \
    --set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled=true

  8. Verify that the management plane pods have a status of Running.

      kubectl get pods -n gloo-mesh --context $MGMT_CONTEXT

    Example output:

      NAME                                      READY   STATUS    RESTARTS   AGE
gloo-mesh-mgmt-server-56c495796b-cx687    1/1     Running   0          30s
gloo-mesh-redis-8455d49c86-f8qhw          1/1     Running   0          30s
gloo-mesh-ui-65b6b6df5f-bf4vp             3/3     Running   0          30s
gloo-telemetry-collector-agent-7rzfb      1/1     Running   0          30s
gloo-telemetry-gateway-6547f479d5-r4zm6   1/1     Running   0          30s
prometheus-server-57cd8c74d4-2bc7f        2/2     Running   0          30s

  9. Save the external address and port that your cloud provider assigned to the gloo-mesh-mgmt-server service. The gloo-mesh-agent agent in each workload cluster accesses this address via a secure connection.

      export MGMT_SERVER_NETWORKING_DOMAIN=$(kubectl get svc -n gloo-mesh gloo-mesh-mgmt-server --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export MGMT_SERVER_NETWORKING_PORT=$(kubectl get svc -n gloo-mesh gloo-mesh-mgmt-server --context $MGMT_CONTEXT -o jsonpath='{.spec.ports[?(@.name=="grpc")].port}')
export MGMT_SERVER_NETWORKING_ADDRESS=${MGMT_SERVER_NETWORKING_DOMAIN}:${MGMT_SERVER_NETWORKING_PORT}
echo $MGMT_SERVER_NETWORKING_ADDRESS
  
      export MGMT_SERVER_NETWORKING_DOMAIN=$(kubectl get svc -n gloo-mesh gloo-mesh-mgmt-server --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
export MGMT_SERVER_NETWORKING_PORT=$(kubectl get svc -n gloo-mesh gloo-mesh-mgmt-server --context $MGMT_CONTEXT -o jsonpath='{.spec.ports[?(@.name=="grpc")].port}')
export MGMT_SERVER_NETWORKING_ADDRESS=${MGMT_SERVER_NETWORKING_DOMAIN}:${MGMT_SERVER_NETWORKING_PORT}
echo $MGMT_SERVER_NETWORKING_ADDRESS

  10. Save the external address and port that your cloud provider assigned to the Gloo OpenTelemetry (OTel) gateway service. The OTel collector agents in each workload cluster send metrics to this address.

      export TELEMETRY_GATEWAY_IP=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export TELEMETRY_GATEWAY_PORT=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.spec.ports[?(@.name=="otlp")].port}')
export TELEMETRY_GATEWAY_ADDRESS=${TELEMETRY_GATEWAY_IP}:${TELEMETRY_GATEWAY_PORT}
echo $TELEMETRY_GATEWAY_ADDRESS
  
      export TELEMETRY_GATEWAY_HOSTNAME=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
export TELEMETRY_GATEWAY_PORT=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.spec.ports[?(@.name=="otlp")].port}')
export TELEMETRY_GATEWAY_ADDRESS=${TELEMETRY_GATEWAY_HOSTNAME}:${TELEMETRY_GATEWAY_PORT}
echo $TELEMETRY_GATEWAY_ADDRESS

Data plane

Register each workload cluster with the Gloo management plane by deploying Gloo data plane components. A deployment named gloo-mesh-agent runs the Gloo agent in each workload cluster.

  1. For the workload cluster that you want to register with Gloo, set the following environment variables. You update these variables each time you follow these steps to register another workload cluster.

      export REMOTE_CLUSTER=<workload_cluster_name>
export REMOTE_CONTEXT=<workload_cluster_context>

  2. In the management cluster, create a KubernetesCluster resource to represent the workload cluster and store relevant data, such as the workload cluster’s local domain.

      kubectl apply --context $MGMT_CONTEXT -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata:
   name: ${REMOTE_CLUSTER}
   namespace: gloo-mesh
spec:
   clusterDomain: cluster.local
EOF

  3. In your workload cluster, apply the Gloo CRDs.

      helm upgrade -i gloo-platform-crds gloo-platform/gloo-platform-crds \
   --namespace=gloo-mesh \
   --create-namespace \
   --version=$GLOO_VERSION \
   --set installEnterpriseCrds=false \
   --kube-context $REMOTE_CONTEXT
  4. Prepare a Helm values file to provide your customizations. To get started, you can use the minimum settings in the following profile as a basis. These settings enable all components that are required for a Gloo data plane installation.
      curl https://storage.googleapis.com/gloo-platform/helm-profiles/$GLOO_VERSION/gloo-core-agent.yaml > data-plane.yaml
open data-plane.yaml

  5. Depending on the method you chose to secure the relay connection, prepare the Helm values for the data plane installation. For more information, see the Setup options.

    In your Helm values file for the workload cluster, add the following values.

      
glooAgent:
  enabled: true
  extraEnvs:
    RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATION:
      value: "true"  
    RELAY_TOKEN: 
      value: "My token"
telemetryCollector:
  enabled: true
telemetryCollectorCustomization:
  skipVerify: true
    Helm valueDescription
    RELAY_TOKENThe relay token to establish initial trust between the Gloo management server and the agent. The relay token is saved in memory on the Gloo agent. You must set the same value that you set in glooMgmtServer.extraEnvs.RELAY_TOKEN.value when you installed the Gloo Network management plane to allow Gloo agents to connect to the Gloo management server.
    RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATIONSet to true to skip validating the server TLS certificate that the Gloo management server presents. This setting is required to configure the relay connection for TLS.
    telemetryCollectorCustomization.skipVerifySet to true to skip validation of the server certificate that the Gloo telemetry gateway presents. By default, the Gloo telemetry gateway uses the same TLS certificates that the Gloo management server uses for the relay connection. If you configure the relay connection for TLS, you must set skipVerify to true on the telemetry collector agent.

    1. Make sure that the following Kubernetes secret exists in the gloo-mesh namespace on each workload cluster.

      • telemetry-root-secret that holds the root CA certificate for the certificate chain

      You can use the steps in BYO server certificate as a guidance for how to create this secret in the workload cluster.

    2. In your Helm values file for the workload cluster, add the following values.

        
glooAgent:
  enabled: true
  extraEnvs:
    RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATION:
      value: "true"  
    RELAY_TOKEN: 
      value: "My token"
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
telemetryCollectorCustomization:
  skipVerify: true
      Helm valueDescription
      RELAY_TOKENThe relay token to establish initial trust between the Gloo management server and the agent. The relay token is saved in memory on the Gloo agent. You must set the same value that you set in glooMgmtServer.extraEnvs.RELAY_TOKEN.value when you installed the Gloo Network management plane to allow Gloo agents to connect to the Gloo management server.
      RELAY_DISABLE_SERVER_CERTIFICATE_VALIDATIONSet to true to skip validating the server TLS certificate that the Gloo management server presents. This setting is required to configure the relay connection for TLS.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.
      telemetryCollectorCustomization.skipVerifySet to true to skip validation of the server certificate that the Gloo telemetry gateway presents. By default, the Gloo telemetry gateway uses the same TLS certificates that the Gloo management server uses for the relay connection. If you configure the relay connection for TLS, you must set skipVerify to true on the telemetry collector agent.

    1. Get the value of the root CA certificate from the management cluster and create a secret in the workload cluster.

        kubectl get secret relay-root-tls-secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl create secret generic relay-root-tls-secret -n gloo-mesh --context $REMOTE_CONTEXT1 --from-file ca.crt=ca.crt
rm ca.crt
  
        kubectl get secret relay-root-tls-secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl create secret generic relay-root-tls-secret -n gloo-mesh --context $REMOTE_CONTEXT2 --from-file ca.crt=ca.crt
rm ca.crt

    2. Get the identity token from the management cluster and create a secret in the workload cluster.

        kubectl get secret relay-identity-token-secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.token}' | base64 -d > token
kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context $REMOTE_CONTEXT1 --from-file token=token
rm token
  
        kubectl get secret relay-identity-token-secret -n gloo-mesh --context $MGMT_CONTEXT -o jsonpath='{.data.token}' | base64 -d > token
kubectl create secret generic relay-identity-token-secret -n gloo-mesh --context $REMOTE_CONTEXT2 --from-file token=token
rm token

    3. Make sure that the following Helm values are added to your values file.

        
glooAgent:
  enabled: true

    1. Make sure that the following Kubernetes secrets exist in the gloo-mesh namespace on each workload cluster.

      • relay-root-tls-secret that holds the root CA certificate
      • relay-identity-token-secret that holds the relay identity token value
      • telemetry-root-secret that holds the root CA certificate for the certificate chain

      You can use the steps in BYO server certificate with managed client certificate as a guidance for how to create these secrets in the workload cluster.

    2. In your Helm values file for the agent, add the following values.

        
glooAgent:
  enabled: true
  relay:
    authority: gloo-mesh-mgmt-server.gloo-mesh
    clientTlsSecretRotationGracePeriodRatio: ""
    rootTlsSecret:
      name: relay-root-tls-secret
      namespace: gloo-mesh
    tokenSecret:
      key: token
      name: relay-identity-token-secret
      namespace: gloo-mesh
telemetryCollector: 
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
      Helm valueDescription
      glooAgent.relay.rootTlsSecretAdd the name and namespace of the Kubernetes secret that holds the root CA credentials that you copied from the management cluster to the workload cluster.
      glooAgent.relay.tokenSecretAdd the name, namespace, and key of the Kubernetes secret that holds the relay identity token that you copied from the management cluster to the workload cluster.
      telemetryCollector.extraVolumesAdd the telemetry-root-secret Kubernetes secret that you created earlier to the root-ca volume. Make sure that you also add the other volumes to your telemetry collector configuration.

    1. Make sure that the following Kubernetes secrets exist in the gloo-mesh namespace on each workload cluster.

      • $REMOTE_CLUSTER1-tls-cert that holds the key, TLS certificate, and certificate chain information for the Gloo agent
      • telemetry-root-secret that holds the root CA certificate for the certificate chain

      You can use the steps in Create certificates with OpenSSL as a guidance for how to create these secrets in the workload cluster.

    2. In your Helm values file for the agent, add the following values. Replace <client-tls-secret-name> with the name of the Kubernetes secret that holds the client TLS certificate, and add the name of the Kubernetes secret that holds the telemetry gateway certificate to the root-ca telemetry collector volume.

        
glooAgent:
  enabled: true
  relay:
    authority: gloo-mesh-mgmt-server.gloo-mesh
    clientTlsSecretRotationGracePeriodRatio: ""
    clientTlsSecret: 
      name: <client-tls-secret-name>
      namespace: gloo-mesh
    tokenSecret:
      key: null
      name: null
      namespace: null
telemetryCollector:
  enabled: true
  extraVolumes: 
    - name: root-ca
      secret:
        defaultMode: 420
        optional: true
        secretName: telemetry-root-secret
    - configMap:
        items:
          - key: relay
            path: relay.yaml
        name: gloo-telemetry-collector-config
      name: telemetry-configmap
    - hostPath:
        path: /var/run/cilium
        type: DirectoryOrCreate
      name: cilium-run
      Helm valueDescription
      glooAgent.relay.clientTlsSecretAdd the name and namespace of the Kubernetes secret that holds the client TLS certificate for the Gloo agent that you created earlier.
      glooAgent.relay.tokenSecretSet all values to null to instruct the Gloo agent to not use identity tokens to establish initial trust with Gloo agents.
      telemetryCollector.extraVolumesAdd the name of the Kubernetes secret that holds the Gloo telemetry gateway certificate that you created earlier to the root-ca volume. Make sure that you also add the configMap and hostPath volumes to your configuration.

  6. Edit the file to provide your own details for settings that are recommended for production deployments, such as the following settings.

    FieldDecription
    glooAgent.resources.limitsAdd resource limits for the gloo-mesh-mgmt-server pod, such as cpu: 500m and memory: 512Mi.
    Cilium observability
    • Cilium metrics: Set glooNetwork.enabled to true to install the Gloo Network-specific agent, which collects additional metrics when Cilium is installed.
    • Cilium logs: To collect Cilium pod logs and view them in the Gloo UI, set telemetryCollectorCustomization.pipelines.logs/cilium_flows.enabled to true. For more information, see Add Cilium flow logs.
    • Cilium insights: To generate insights that analyze your Cilium setup’s health, set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled to true. This pipeline uses the default filter/cilium processor that collects all Cilium and Hubble metrics. You can optionally customize the Cilium metrics collection to reduce the number of metrics that are collected.

  7. Use the customizations in your Helm values file to install the Gloo data plane components in your workload cluster.

      helm upgrade -i gloo-platform gloo-platform/gloo-platform \
    --kube-context $REMOTE_CONTEXT \
    -n gloo-mesh \
    --version $GLOO_VERSION \
    --values data-plane.yaml \
    --set common.cluster=$REMOTE_CLUSTER \
    --set glooAgent.relay.serverAddress=$MGMT_SERVER_NETWORKING_ADDRESS \
    --set glooNetwork.enabled=true \
    --set telemetryCollector.config.exporters.otlp.endpoint=$TELEMETRY_GATEWAY_ADDRESS \
    --set telemetryCollectorCustomization.pipelines.logs/cilium_flows.enabled=true \
    --set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled=true

  8. Verify that the Gloo data plane component pods are running. If not, try debugging the agent.

      meshctl check --kubecontext $REMOTE_CONTEXT

    Example output:

      🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-agent                | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 3/3   | Healthy

  9. Repeat steps 1 - 8 to register each workload cluster with Gloo.

  10. Verify that your Gloo Network setup is correctly installed. If not, try debugging the relay connection. Note that this check might take a few seconds to verify that:

    • Your Gloo product licenses are valid and current.
    • The Gloo CRDs are installed at the correct version.
    • The management plane pods in the management cluster are running and healthy.
    • The agents in the workload clusters are successfully identified by the management server.
      meshctl check --kubecontext $MGMT_CONTEXT

    Example output:

      🟢 License status

INFO  gloo-network enterprise license expiration is 25 Aug 24 10:38 CDT
INFO  No GraphQL license module found for any product

🟢 CRD version check

🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-mgmt-server          | 1/1   | Healthy
gloo-mesh | gloo-mesh-redis                | 1/1   | Healthy
gloo-mesh | gloo-mesh-ui                   | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 3/3   | Healthy
gloo-mesh | gloo-telemetry-gateway         | 1/1   | Healthy
gloo-mesh | prometheus-server              | 1/1   | Healthy

🟢 Mgmt server connectivity to workload agents

Cluster  | Registered | Connected Pod                                   
cluster1 | true       | gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6
cluster2 | true       | gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6

Connected Pod                                    | Clusters
gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6 | 2

Next steps

Now that you have Gloo Network for Cilium up and running, check out some of the following resources to learn more about Gloo Network and expand your Cilium capabilities.

Cilium:

Gloo Network:

  • Explore insights to review and improve your setup’s health and security posture.
  • When it’s time to upgrade Gloo Network, see the upgrade guide.

Istio: To also deploy Istio in your environment, check out Gloo Mesh Core to quickly install and manage your service mesh for you with service mesh lifecycle management.

Help and support: