☰
Navigation :

Best practices for production

Review the following recommended practices for preparing optional security measures and setting up Gloo Mesh Enterprise in a production environment.

Deployment model

A Gloo Mesh setup consists of one management cluster that the Gloo Mesh Enterprise management components are installed in, and one or more workload clusters that run services meshes which are registered with and managed by the management cluster. The management cluster serves as the management plane, and the workload clusters serve as the data plane.

In a production deployment, do not install the management components into a workload cluster that runs a service mesh. Instead, use a dedicated cluster for the management plane, in which you will create Gloo Mesh resources to configure and manage service meshes in data plane clusters.

Figure of a multicluster Gloo Mesh quick-start architecture, with a dedicated management plane cluster.

Management plane settings

Before you install the Gloo Mesh management components into your management cluster, review the following options to help secure your installation. Each section details the benefits of the security option, and the necessary settings to specify in a Helm values file to use during your Helm installation.

The settings in the following sections are also provided in the sample values-mgmt-plane.yaml values file.

FIPS-compliant image

If your environment runs workloads that require federal information processing compliance, you can use images of Gloo Mesh Enterprise components that are specially built to comply with NIST FIPS. Open the values.yaml file, search for the image section, and append -fips to the tag, such as in the following example. 
...
enterpriseNetworking:
  image:
    pullPolicy: IfNotPresent
    registry: gcr.io/gloo-mesh
    repository: enterprise-networking
    tag: 1.2.26-fips

Certificate management

Gloo Mesh Enterprise's default behavior is to create self-signed certificates at install time to handle bootstrapping mTLS connectivity between the relay server and agent components of Gloo Mesh Enterprise. To use these default certificates, leave the selfSigned value set to true by default. If you prefer to set up Gloo Mesh without secure communication for quick demonstrations, include the --set enterprise-networking.global.insecure=true flag. Note that using the default self-signed certificate authorities (CAs) or using insecure mode are not suitable for production environments.

In production installations, do not use the default root CA certificate and intermediate signing CAs that are automatically generated and self-signed by Gloo Mesh. Instead, add automation so that the certificates can be easily rotated as described in the certificate management guide.

To supply your custom certificates during Gloo Mesh installation:

  1. Select the certificate management approach that you want to use, such as AWS Certificate Manager, HashiCorp Vault, or your own custom certs.
  2. As you follow those instructions, make sure that you create relay forwarding and identity secrets in the management and remote clusters.
  3. As you follow those instructions, modify your Helm values file to use the custom CAs, such as in the following enterprise-networking section. Note that you might need to update the cluster and relayTlsSecret name values, depending on your certificate setup.
enterprise-networking:
  # Set to true to permit unencrypted and unauthenticated communication between management plane and data planes.
  global:
    insecure: false
  # Name of the management cluster.
  cluster: $MGMT_CLUSTER

  # Custom certs: Reference to a secret containing TLS certificates used to secure the networking gRPC server with TLS.
  relayTlsSecret:
    name: enterprise-networking-server-certs
    namespace: gloo-mesh

  # Create self-signed CA certificate and bootstrap token secrets for the relay server.
  selfSigned: false
  
  # Disable default relay CA functionality only when supplying custom client certs to the agents for relay mTLS.
  disableRelayCa: true

Dashboard authentication

The Gloo Mesh dashboard supports OpenID Connect (OIDC) authentication from common providers such as Google, Okta, and Auth0. Users that access the dashboard will be required to authenticate with the OIDC provider, and all requests to retrieve data from the API will be authenticated.

You can configure OIDC authentication for the dashboard by providing your OIDC provider details in a gloo-mesh-ui section, such as the following.

...
gloo-mesh-ui:
  dashboard:
    enabled: true
  auth:
    enabled: true
    backend: oidc
    oidc:
      clientId: # From the OIDC provider
      clientSecret: # From the OIDC provider (will be stored in secret)
      clientSecretRef:
        name: dashboard
        namespace: gloo-mesh
      issuerUrl: # The issuer URL from the OIDC provider, such as https://<domain>.<provider url>/
      appUrl: # URL the dashboard is available at from DNS and other ingress settings that expose the dashboard service

For more information about the available settings for the dashboard Helm chart, see the reference for the dashboard Helm values.

RBAC

Gloo Mesh's role-based API allows you to restrict access to policy configuration based on the roles of individual users, represented by a Role CRD. When the role-based API is enabled, an implicit deny on all policy actions is created. Similar to the Kubernetes RBAC model, Gloo Mesh users must be bound to one or more roles that grant them the permissions they need to configure policies in Gloo Mesh.

You can enable and configure the role-based API by providing your OIDC provider details in a gloo-mesh-ui section, such as the following.

...
rbac-webhook:
  enabled: true
  # Assign users or groups to the admin role
  adminSubjects:
  - kind: User
    name: kubernetes-admin
  # Create a Kubernetes role for Gloo Mesh admins
  createAdminRole: true

  # Deployment configuration for rbac-webhook app
  rbacWebhook:
    enabled: true
    image:
      pullPolicy: IfNotPresent
      registry: gcr.io/gloo-mesh
      repository: rbac-webhook
      tag: 1.2.26
    ports:
      webhook: 8443
    # rbac-webhook pod resources
    resources:
      requests:
        cpu: 125m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Gi
    # Kubernetes service type
    serviceType: ClusterIP

Data plane settings

Before you register remote clusters with Gloo Mesh, review the following options to help secure your registration. Each section details the benefits of the security option, and the necessary settings to specify in a Helm values file to use during your Helm registration.

The settings in the following sections are also provided in the sample values-data-plane.yaml values file.

FIPS-compliant image

If your environment runs workloads that require federal information processing compliance, you can use images of Gloo Mesh Enterprise components that are specially built to comply with NIST FIPS. Open the values.yaml file, search for the image section, and append -fips to the tag, such as in the following example. 
...
enterpriseAgent:
  image:
    pullPolicy: IfNotPresent
    registry: gcr.io/gloo-mesh
    repository: enterprise-agent
    tag: 1.2.26-fips

Certificate management

If you use the default self-signed certificates during Gloo Mesh installation, you can follow the steps in the cluster registration documentation to use these certificates during cluster registration. If you set up Gloo Mesh without secure communication for quick demonstrations, include the --set global.insecure=true flag during registration. Note that using the default self-signed certificate authorities (CAs) or using insecure mode are not suitable for production environments.

In production environments, you use the same custom certificates that you set up for Gloo Mesh installation during cluster registration:

  1. Ensure that when you installed Gloo Mesh, you set up the relay certificates, such as with AWS Certificate Manager, HashiCorp Vault, or your own custom certs, including the relay forwarding and identity secrets in the management and remote clusters.
  2. The relay certificate instructions include steps to modify your Helm values file to use the custom CAs, such as in the following relay section. Note that you might need to update the cluster, serverAddress, clientCertSecret name, and rootTlsSecret name values, depending on your certificate setup.
relay:
  # SNI name used to connect to relay forwarding server, which must match server certificate CommonName (DO NOT CHANGE)
  authority: enterprise-networking.gloo-mesh
  # Remote cluster name
  cluster: $REMOTE_CLUSTER
  # management plane
  serverAddress: $RELAY_ADDRESS:9090
  # Set to true to permit unencrypted and unauthenticated communication to the management plane
  insecure: false
  # Reference to a secret containing a root TLS cert used to verify the relay server cert. 
  # The secret can also optionally specify a 'tls.key' which will be used to generate the agent client cert.
  rootTlsSecret:
    name: relay-root-tls-secret
    namespace: gloo-mesh
  # Reference to a secret containing TLS certs used to secure the networking gRPC server with TLS.
  clientCertSecret:
    name: $REMOTE_CLUSTER-relay-certs
    namespace: gloo-mesh
  # Enable verbose/debug logging
  verbose: false 
rate-limiter: 
  enabled: false
ext-auth-service: 
  enabled: false
enterpriseAgent:
  enabled: true

Rate limiting and external authentication

To enable mTLS with rate limiting and external authentication, you must add an injection directive for those components. Although you can enable an injection directive on the gloo-mesh namespace, this directive makes the management plane components dependent on the functionality of Istio’s mutating webhook, which may be a fragile coupling and is not recommended as best practice. In production setups, install the Gloo Mesh Enterprise chart with just rate limiting and external authentication services enabled to the gloo-mesh-addons namespace, and label the gloo-mesh-addons namespace for Istio injection.

When you initially register a cluster, set the rate-limiter and ext-auth-service settings to false in the values file, such as the following.

rate-limiter: 
  enabled: false
ext-auth-service: 
  enabled: false
enterpriseAgent:
  enabled: true

After registration is complete, you can then follow the steps in Setting up rate limiting and external authentication to create the gloo-mesh-addons namespace and enable rate limiting and external authentication in that namespace.