Use Gloo Network for Cilium to provide connectivity, security, and observability for containerized workloads with a Cilium-based container network interface (CNI) plug-in that leverages the Linux kernel technology eBPF. Gloo Network additionally provides insights into your Cilium environment through a custom dashboard.

You can follow this guide to quickly get started with Gloo Network. To learn more about the benefits and architecture, see About. To customize your installation with Helm instead, see the advanced installation guide.

Before you begin

  1. Install the following command-line (CLI) tools.

    • kubectl, the Kubernetes command line tool. Download the kubectl version that is within one minor version of the Kubernetes clusters you plan to use.
    • meshctl, the Solo command line tool.
        curl -sL https://run.solo.io/meshctl/install | GLOO_MESH_VERSION=v2.5.6 sh -
export PATH=$HOME/.gloo-mesh/bin:$PATH
    • helm, the Kubernetes package manager.
    • cilium, the Cilium command line tool.

  2. Create environment variables for the following details.

    • GLOO_NETWORK_LICENSE_KEY: Set your Gloo Network for Cilium license key as an environment variable. If you do not have one, contact an account representative.
    • GLOO_VERSION: Set the Gloo Network version. This example uses the latest version. Append -fips for a FIPS-compliant image. Do not include v before the version number.
    • SOLO_CILIUM_REPO: A repo key for the Solo distribution of Cilium that you can get by logging in to the Support Center and reviewing the Cilium images built by Solo.io support article.
    • CILIUM_VERSION: The Cilium version that you want to install. This example uses the latest version.
      export GLOO_NETWORK_LICENSE_KEY=<license_key>
export GLOO_VERSION=2.5.6
export SOLO_CILIUM_REPO=<cilium_repo_key>
export CILIUM_VERSION=1.14.2

  3. Optional: If you plan to run Istio with sidecar injection and the Cilium CNI in tunneling mode (VXLAN or GENEVE) on an Amazon EKS cluster, see Considerations for running Cilium and Istio on EKS.

Install the Solo distribution of the Cilium CNI

  1. Create or use Kubernetes clusters that meet the Cilium requirements. For example, to try out the Cilium CNI in Google Kubernetes Engine (GKE) clusters, your clusters must be created with specific node taints.

    1. Open the Cilium documentation and find the cloud provider that you want to use to create your clusters.

    2. Follow the steps of your cloud provider to create one or more clusters that meet the Cilium requirements.

      • The instructions in the Cilium documentation might create a cluster with insufficient CPU and memory resources for Gloo Network. Make sure that you use a machine type with at least 2vCPU and 8GB of memory.
      • The cluster name must be alphanumeric with no special characters except a hyphen (-), lowercase, and begin with a letter (not a number).
      • Multicluster setups only: For a multicluster setup, you need at least two clusters. One cluster is set up as the Gloo management plane where the management components are installed. The other cluster is registered as your data plane and runs your Kubernetes workloads and Istio service mesh. You can optionally add more workload clusters to your setup. The instructions in this guide assume one management cluster and two workload clusters.

      Example to create a cluster in GKE:

        export NAME="$(whoami)-$RANDOM"                                                        
gcloud container clusters create "${NAME}" \
--node-taints node.cilium.io/agent-not-ready=true:NoExecute \
--zone us-west2-a \
--machine-type e2-standard-2
gcloud container clusters get-credentials "${NAME}" --zone us-west2-a

  2. Add and update the Cilium Helm repo.

      helm repo add cilium https://helm.cilium.io/
helm repo update

  3. Install the CNI by using a Solo distribution of Cilium in your cluster. Be sure to include the following settings for compatability with Gloo Network, and the necessary settings for your infrastructure provider.

    Generic example (add cloud provider-specific values in --set flags below):

      helm install cilium cilium/cilium \
--namespace kube-system \
--version ${CILIUM_VERSION} \
--set customCalls.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enabled="{dns:destinationContext=pod;sourceContext=pod,drop:destinationContext=pod;sourceContext=pod,tcp:destinationContext=pod;sourceContext=pod,flow:destinationContext=pod;sourceContext=pod,port-distribution:destinationContext=pod;sourceContext=pod}" \
--set image.repository=${SOLO_CILIUM_REPO}/cilium \
--set image.useDigest=false \
--set operator.image.repository=${SOLO_CILIUM_REPO}/operator \
--set operator.image.useDigest=false \
--set operator.prometheus.enabled=true \
--set prometheus.enabled=true \
# ADD CLOUD PROVIDER-SPECIFIC HELM VALUES HERE

    Example for GKE:

      NATIVE_CIDR="$(gcloud container clusters describe "${NAME}" --zone "${ZONE}" --format 'value(clusterIpv4Cidr)')"
echo $NATIVE_CIDR

helm install cilium cilium/cilium \
--namespace kube-system \
--version ${CILIUM_VERSION} \
--set customCalls.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enabled="{dns:destinationContext=pod;sourceContext=pod,drop:destinationContext=pod;sourceContext=pod,tcp:destinationContext=pod;sourceContext=pod,flow:destinationContext=pod;sourceContext=pod,port-distribution:destinationContext=pod;sourceContext=pod}" \
--set image.repository=${SOLO_CILIUM_REPO}/cilium \
--set image.useDigest=false \
--set operator.image.repository=${SOLO_CILIUM_REPO}/operator \
--set operator.image.useDigest=false \
--set operator.prometheus.enabled=true \
--set prometheus.enabled=true \
--set nodeinit.enabled=true \
--set nodeinit.reconfigureKubelet=true \
--set nodeinit.removeCbrBridge=true \
--set cni.binPath=/home/kubernetes/bin \
--set gke.enabled=true \
--set ipam.mode=kubernetes \
--set ipv4NativeRoutingCIDR=${NATIVE_CIDR}

    Generic example (add any cloud provider-specific values):

      helm upgrade -i -n kube-system cilium cilium/cilium --version ${CILIUM_VERSION} -f - <<EOF
customCalls:
  enabled: true
hubble:
  enabled: true
  metrics:
    enabled:
    - dns:destinationContext=pod;sourceContext=pod
    - drop:destinationContext=pod;sourceContext=pod
    - tcp:destinationContext=pod;sourceContext=pod
    - flow:destinationContext=pod;sourceContext=pod
    - port-distribution:destinationContext=pod;sourceContext=pod
image:
  repository: ${SOLO_CILIUM_REPO}/cilium
  useDigest: false
operator:
  replicas: 1
  prometheus:
    enabled: true
  image:
    repository: ${SOLO_CILIUM_REPO}/operator
    useDigest: false
prometheus:
  enabled: true
# ADD CLOUD PROVIDER-SPECIFIC HELM VALUES
EOF

    Example for GKE:

      NATIVE_CIDR="$(gcloud container clusters describe "${NAME}" --zone "${ZONE}" --format 'value(clusterIpv4Cidr)')"
echo $NATIVE_CIDR

helm upgrade -i -n kube-system cilium cilium/cilium --version ${CILIUM_VERSION} -f - <<EOF
customCalls:
  enabled: true
hubble:
  enabled: true
  metrics:
    enabled:
    - dns:destinationContext=pod;sourceContext=pod
    - drop:destinationContext=pod;sourceContext=pod
    - tcp:destinationContext=pod;sourceContext=pod
    - flow:destinationContext=pod;sourceContext=pod
    - port-distribution:destinationContext=pod;sourceContext=pod
image:
  repository: ${SOLO_CILIUM_REPO}/cilium
  useDigest: false
operator:
  replicas: 1
  prometheus:
    enabled: true
  image:
    repository: ${SOLO_CILIUM_REPO}/operator
    useDigest: false
prometheus:
  enabled: true
nodeinit:
  enabled: true
  reconfigureKubelet: true
  removeCbrBridge: true
cni:
  binPath: /home/kubernetes/bin
gke:
  enabled: true
ipam:
  mode: kubernetes
ipv4NativeRoutingCIDR: ${NATIVE_CIDR}
EOF

    Example output:

      NAME: cilium
LAST DEPLOYED: Fri Sep 16 10:31:52 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.14.2.

For any further help, visit https://docs.cilium.io/en/v1.12/gettinghelp

  4. Verify that the Cilium CNI is successfully installed. Because the Cilium agent is deployed as a daemon set, the number of Cilium and Cilium node init pods equals the number of nodes in your cluster.

      kubectl get pods -n kube-system | grep cilium

    Example output:

      cilium-gbqgq                                                  1/1     Running             0          48s
cilium-j9n5x                                                  1/1     Running             0          48s
cilium-node-init-c7rxb                                        1/1     Running             0          48s
cilium-node-init-pnblb                                        1/1     Running             0          48s
cilium-node-init-wdtjm                                        1/1     Running             0          48s
cilium-operator-69dd4567b5-2gjgg                              1/1     Running             0          47s
cilium-operator-69dd4567b5-ww6wp                              1/1     Running             0          47s
cilium-smp9c                                                  1/1     Running             0          48s

  5. Check the status of the Cilium installation.

      cilium status --wait

    Example output:

      
____/¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet              cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:            cilium-operator    Running: 2
                       cilium             Running: 4
Cluster Pods:          3/3 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             ${SOLO_CILIUM_REPO}/cilium:v1.14.2: 4
                       cilium-operator    ${SOLO_CILIUM_REPO}/operator-generic:v1.14.2: 2

  6. Multicluster setups only: Repeat steps 3 - 5 to install the CNI in each cluster that you want to use in your Gloo Network environment.

Install Gloo Network for Cilium

In a multicluster setup, you deploy the Gloo management plane into a dedicated management cluster, and the Gloo data plane into one or more workload clusters.

Management plane

Deploy the Gloo management plane into a dedicated management cluster.

  1. Set the names of your clusters from your infrastructure provider. If your clusters have different names, specify those names instead.

      export MGMT_CLUSTER=mgmt
export REMOTE_CLUSTER1=cluster1
export REMOTE_CLUSTER2=cluster2
  2. Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The generated certificate that connects workload clusters to the management cluster uses the context name as a SAN specification, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
      export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT1=<remote-cluster1-context>
export REMOTE_CONTEXT2=<remote-cluster2-context>

  3. Install Gloo Network in your management cluster. This command uses a basic profile to create a gloo-mesh namespace and install the Gloo management plane components, such as the management server and Prometheus server, in your management cluster. For more information, check out the CLI install profiles.

      meshctl install --profiles gloo-core-mgmt,cilium-flows \
  --kubecontext $MGMT_CONTEXT \
  --set common.cluster=$MGMT_CLUSTER \
  --set licensing.glooNetworkLicenseKey=$GLOO_NETWORK_LICENSE_KEY \
  --set telemetryCollectorCustomization.pipelines.metrics/cilium.enabled=true

  4. Verify that the management plane pods have a status of Running.

      kubectl get pods -n gloo-mesh --context $MGMT_CONTEXT

    Example output:

      NAME                                      READY   STATUS    RESTARTS   AGE
gloo-mesh-mgmt-server-56c495796b-cx687    1/1     Running   0          30s
gloo-mesh-redis-8455d49c86-f8qhw          1/1     Running   0          30s
gloo-mesh-ui-65b6b6df5f-bf4vp             3/3     Running   0          30s
gloo-telemetry-collector-agent-7rzfb      1/1     Running   0          30s
gloo-telemetry-gateway-6547f479d5-r4zm6   1/1     Running   0          30s
prometheus-server-57cd8c74d4-2bc7f        2/2     Running   0          30s

Data plane

Register each workload cluster with the Gloo management plane by deploying Gloo data plane components. A deployment named gloo-mesh-agent runs the Gloo agent in each workload cluster.

  1. Save the external address and port that your cloud provider assigned to the Gloo OpenTelemetry (OTel) gateway service. The OTel collector agents in each workload cluster send metrics to this address.

      export TELEMETRY_GATEWAY_IP=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export TELEMETRY_GATEWAY_PORT=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.spec.ports[?(@.name=="otlp")].port}')
export TELEMETRY_GATEWAY_ADDRESS=${TELEMETRY_GATEWAY_IP}:${TELEMETRY_GATEWAY_PORT}
echo $TELEMETRY_GATEWAY_ADDRESS
  
      export TELEMETRY_GATEWAY_HOSTNAME=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
export TELEMETRY_GATEWAY_PORT=$(kubectl get svc -n gloo-mesh gloo-telemetry-gateway --context $MGMT_CONTEXT -o jsonpath='{.spec.ports[?(@.name=="otlp")].port}')
export TELEMETRY_GATEWAY_ADDRESS=${TELEMETRY_GATEWAY_HOSTNAME}:${TELEMETRY_GATEWAY_PORT}
echo $TELEMETRY_GATEWAY_ADDRESS

  2. Save the following settings in a values file.

      cat > "cilium-values.yaml" <<EOF
telemetryCollectorCustomization:
  pipelines:
    metrics/cilium:
      enabled: true
EOF

  3. Register both workload clusters with the management server. These commands use basic profiles to create a gloo-mesh namespace and install the Gloo data plane components, such as the Gloo agent. For more information, check out the CLI install profiles.

      meshctl cluster register $REMOTE_CLUSTER1 \
  --kubecontext $MGMT_CONTEXT \
  --profiles gloo-core-agent,cilium-flows \
  --remote-context $REMOTE_CONTEXT1 \
  --telemetry-server-address $TELEMETRY_GATEWAY_ADDRESS \
  --gloo-mesh-agent-chart-values cilium-values.yaml

meshctl cluster register $REMOTE_CLUSTER2 \
  --kubecontext $MGMT_CONTEXT \
  --profiles gloo-core-agent,cilium-flows \
  --remote-context $REMOTE_CONTEXT2 \
  --telemetry-server-address $TELEMETRY_GATEWAY_ADDRESS \
  --gloo-mesh-agent-chart-values cilium-values.yaml

  4. Verify that the Gloo data plane components in each workload cluster are healthy. If not, try debugging the agent.

      meshctl check --kubecontext $REMOTE_CONTEXT1
meshctl check --kubecontext $REMOTE_CONTEXT2

    Example output:

      🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-agent                | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 3/3   | Healthy

  5. Verify that your Gloo Network setup is correctly installed. If not, try debugging the relay connection. Note that this check might take a few seconds to verify that:

    • Your Gloo product licenses are valid and current.
    • The Gloo CRDs are installed at the correct version.
    • The management plane pods in the management cluster are running and healthy.
    • The agents in the workload clusters are successfully identified by the management server.
      meshctl check --kubecontext $MGMT_CONTEXT

    Example output:

      🟢 License status

INFO  gloo-network enterprise license expiration is 25 Aug 24 10:38 CDT
INFO  No GraphQL license module found for any product

🟢 CRD version check

🟢 Gloo deployment status

Namespace | Name                           | Ready | Status
gloo-mesh | gloo-mesh-mgmt-server          | 1/1   | Healthy
gloo-mesh | gloo-mesh-redis                | 1/1   | Healthy
gloo-mesh | gloo-mesh-ui                   | 1/1   | Healthy
gloo-mesh | gloo-telemetry-collector-agent | 3/3   | Healthy
gloo-mesh | gloo-telemetry-gateway         | 1/1   | Healthy
gloo-mesh | prometheus-server              | 1/1   | Healthy

🟢 Mgmt server connectivity to workload agents

Cluster  | Registered | Connected Pod                                   
cluster1 | true       | gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6
cluster2 | true       | gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6

Connected Pod                                    | Clusters
gloo-mesh/gloo-mesh-mgmt-server-65bd557b95-v8qq6 | 2

Explore the UI

Visualize and analyze your Cilium and Gloo Network setup by deploying sample apps and launching the Gloo UI.

Launch the dashboard

  1. Open the Gloo UI. The Gloo UI is served from the gloo-mesh-ui service on port 8090. You can connect by using the meshctl or kubectl CLIs.

    For more information, see the CLI documentation.

      meshctl dashboard --context $MGMT_CONTEXT
    1. Port-forward the gloo-mesh-ui service on 8090.
        kubectl port-forward -n gloo-mesh --context $MGMT_CONTEXT svc/gloo-mesh-ui 8090:8090
    2. Open your browser and connect to http://localhost:8090.

  2. Review your Dashboard for an at-a-glance overview of your Gloo Network environment. Environment insights, health, status, inventories, and more are summarized in the following cards:

    • Analysis and Insights: Gloo Network recommendations for how to improve your Cilium setups.
    • Cilium and Gloo health: A status check of the Gloo Network and Cilium installations in your clusters.
    • Cluster Services: Inventory of services across all clusters in your Gloo Network setup, and whether those services are in a service mesh or not.


    Figure: Gloo UI dashboard
    Figure: Gloo UI dashboard

  3. Click the Cilium tab in the Cilium and Gloo health card. Verify that the Cilium version in each cluster was discovered.

    Cilium installation health card in the Gloo UI

Check insights

Review the insights for your environment. Gloo Network comes with an insights engine that automatically analyzes your Cilium setup for health issues. Then, Gloo shares these issues along with recommendations to harden your Cilium setup. The insights give you a checklist to address issues that might otherwise be hard to detect across your environment.

  1. On the Analysis and Insights card of the dashboard, you can quickly see a summary of the insights for your environment, including how many insights are available at each severity level, and the type of insight.

    Figure: Insights and analysis card
    Figure: Insights and analysis card

  2. View the list of insights by clicking the Details button, or go to the Insights page.

  3. On the Insights page, you can view recommendations to harden your Cilium setup, and steps to implement them in your environment. Gloo Network analyzes your setup, and returns individual insights that contain information about errors and warnings in your environment, best practices you can use to improve your configuration and security, and more.

    Figure: Insights page
    Figure: Insights page

  4. On an insight that you want to resolve, click Details. The details modal shows more data about the insight, such as the time when it was last observed in your environment, and if applicable, the extended settings or configuration that the insight applies to.

  5. Click the Target YAML tab to see the resource file that the insight references, and click the View Resolution Steps tab to see guidance such as steps for fixing warnings and errors in your resource configuration or recommendations for improving your security and setup.

Apply a Cilium network policy

Deploy a demo app and try out a Cilium network policy to secure and control traffic flows between app microservices.

  1. Deploy the Cilium Star Wars demo app in your cluster.

    1. Create a namespace for the demo app.

        kubectl create ns starwars

    2. Deploy the demo app, which includes tiefighter, xwing, and deathstar pods, and a deathstar service. The tiefighter and deathstar pods have the org=empire label, and the xwing pod has the org=alliance label.

        kubectl -n starwars apply -f https://raw.githubusercontent.com/cilium/cilium/$CILIUM_VERSION/examples/minikube/http-sw-app.yaml

    3. Verify that the demo pods and service are running.

        kubectl get pods,svc -n starwars

      Example output:

        NAME                             READY   STATUS    RESTARTS   AGE
pod/deathstar-6fb5694d48-5hmds   1/1     Running   0          107s
pod/deathstar-6fb5694d48-fhf65   1/1     Running   0          107s
pod/tiefighter                   1/1     Running   0          107s
pod/xwing                        1/1     Running   0          107s

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/deathstar    ClusterIP   10.96.110.8   <none>        80/TCP    107s

  2. Generate some network traffic by sending requests from the xwing and tiefighter pods to the deathstar service.

      kubectl exec xwing -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing
kubectl exec tiefighter -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing

    Example output for both commands:

      Ship landed
Ship landed

  3. Create a Cilium network policy that allows only apps that have the org=empire label to access the deathstar app. After you create this access policy, only the tiefighter pod can access the deathstar app.

      kubectl apply -f - << EOF
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
  namespace: starwars
spec:
  description: "L3-L4 policy to restrict deathstar access to empire ships only"
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
EOF

  4. Send another request from the tiefighter pod to the deathstar service.

      kubectl exec tiefighter -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing

    The request succeeds, because requests from apps with the org=empire label are permitted. Example output:

      Ship landed

  5. Send another request from the xwing pod to the deathstar service.

      kubectl exec xwing -n starwars -- curl -s -XPOST deathstar.starwars.svc.cluster.local/v1/request-landing

    This request hangs, because requests from apps without the org=empire label are not permitted. No Layer 7 HTTP response code is returned, because the request is dropped on Layer 3. You can enter control+c to stop the curl request, or wait for it to time out.

  6. You can also check the metrics to verify that the policy allowed or blocked requests.

    1. Open the Prometheus expression browser.
      • meshctl: For more information, see the CLI documentation.
          meshctl proxy prometheus
      • kubectl:
        1. Port-forward the prometheus-server deployment on 9091.
            kubectl -n gloo-mesh port-forward deploy/prometheus-server 9091
        2. Open your browser and connect to localhost:9091/.
    2. In the Expression search bar, paste the following query and click Execute.
        rate(hubble_drop_total{destination_workload_id=~"deathstar.+"}[5m])
    3. Verify that you can see requests from the xwing pod to the deathstar service that were dropped because of the network policy.

Next steps

Now that you have Gloo Network for Cilium up and running, check out some of the following resources to learn more about Gloo Network and expand your Cilium capabilities.

Cilium:

Gloo Network:

  • Customize your Gloo Network installation with a Helm-based setup.
    • Explore insights to review and improve your setup’s health and security posture.
    • When it’s time to upgrade Gloo Network, see the upgrade guide.

    Istio: To also deploy Istio in your environment, check out Gloo Mesh Core to quickly install and manage your service mesh for you with service mesh lifecycle management.

    Help and support:

    Cleanup

    1. Remove the demo app resources and namespace.

        kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/$CILIUM_VERSION/examples/minikube/http-sw-app.yaml -n starwars
kubectl delete cnp rule1 -n starwars
kubectl delete ns starwars

    2. If you no longer need this quick-start Gloo Network environment, you can uninstall the Gloo management and agent components by following the steps in the uninstall guide.