Installing Gloo Gateway on HashiCorp Nomad

Gloo Gateway can be used as an Ingress/Gateway for the Nomad platform. This guide walks through the process of installing Gloo on Nomad, using Consul for service discovery/configuration and Vault for secret storage.

HashiCorp Nomad is a popular workload scheduler that can be used in place of, or in combination with Kubernetes as a way of running long-lived processes on a cluster of hosts. Nomad supports native integration with Consul and Vault, making configuration, service discovery, and credential management easy for application developers.

You can see a demonstration of Gloo using Consul, Nomad, and Vault in this YouTube video.


Architecture

Gloo Gateway on Nomad Architecture

Gloo Gateway on Nomad uses multiple pieces of software for deployment and functionality.

Preparing for Installation

Before proceeding to the installation, you will need to complete some prerequisites.

Prerequisite Software

Installation on Nomad requires the following:

Download the Installation Files

This tutorial uses files stored on the Gloo GitHub repository.

In order to install Gloo on Nomad, you’ll want to clone the repository:

git clone https://github.com/solo-io/gloo
cd gloo/install/nomad

The files used for installation live in the install/nomad directory.

├── demo.sh
├── gloo-policy.hcl
├── jobs
│   ├── gloo.nomad
│   └── petstore.nomad
├── launch-consul-vault-nomad-dev.sh
├── README.md
├── Vagrantfile
└── variables
    ├── variables-linux.yaml
    └── variables-mac.yaml

The Gloo Nomad Job and the Pet Store job are in the jobs directory.

The gloo.nomad job is experimental and designed to be used with a specific Vault + Consul + Nomad setup.

The Levant Variables for the Gloo Nomad Job are in the variables directory.

Inputs for the job can be tweaked by modifying variables/variables-*.yaml files.


Deploying Gloo with Nomad

The scripts and files included in the Gloo repository provide three different options for deployment:

Run the complete Demo

If your environment is set up with Docker, Nomad, Consul, Vault, and Levant, you can simply run demo.sh to create a local demo of Gloo routing to the PetStore Nomad. The script will spin up dev instances of Consult, Nomad, and Vault. Then it will use Nomad to deploy the Gloo Gateway and the Pet Store application. Finally, it will create a route on the Gloo Gateway to the Pet Store application.

./demo.sh

After the script completes its setup process, you can test out the routing rule on Gloo by running the following command.

curl <nomad-host>:8080/

If running on macOS or with Vagrant:

curl localhost:8080/

If running on Linux, use the Host IP on the docker0 interface:

curl 172.17.0.1:8080/

The value returned should be:

[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]

Running Nomad Using Vagrant

The provided Vagrantfile will run Nomad, Consul, and Vault inside a VM on your local machine.

First download and install HashiCorp Vagrant.

Then run the following command:

vagrant up

Ports will be forwarded to your local system, allowing you to access services on the following ports (on localhost):

service port
nomad 4646
consul 8500
vault 8200
gloo/http 8080
gloo/https 8443
gloo/admin 19000

Running Nomad, Consul, and Vault

If you’ve installed Nomad/Consul/Vault locally, you can use launch-consul-vault-nomad-dev.sh to run them on your local system.

If running locally (without Vagrant) on macOS, you will need to install the Weave Net Docker Plugin:

docker swarm init # if your docker host is not currently a swarm manager
docker plugin install weaveworks/net-plugin:latest_release --grant-all-permissions
docker plugin enable weaveworks/net-plugin:latest_release
docker network create --driver=weaveworks/net-plugin:latest_release --attachable weave

If running locally on Linux, you’ll need to disable SELinux in order to run the demo (or add permission for docker containers to access / on their filesystem):

sudo setenforce 0

Then run the launch-consul-vault-nomad-dev.sh script.

./launch-consul-vault-nomad-dev.sh

The script will launch a dev instance of Consul, Vault, and Nomad and then continue to monitor the status of those services in debug mode. You can stop all of the services by hitting Ctrl-C.

Once you have finished launching these services, you are now ready to install Gloo on either your Linux or macOS system.


Installing Gloo on Nomad

Once you have a base environment set up with Consul, Vault, and Nomad running, you are ready to deploy the Nomad job that creates the necessary containers to run Gloo. The next two sections will guide you on installing Gloo on Linux or macOS.

Installing Gloo on Nomad (Linux)

In this step we will deploy Gloo using Levant on a Linux-based system. The assumption is that you are running Consul, Nomad, and Vault either locally or remotely.

If you are running these services remotely, then you will need to update the address and consul-address values with your configuration. The default port for Nomad is 4646 and for Consul is 8500. Make sure to give the full address to your Nomad and Consul servers, e.g. https://my.consul.local:8500.

levant deploy \
    -var-file variables/variables-linux.yaml \
    -address http://<nomad-host>:<nomad-port> \
    -consul-address http://<consul-host>:<consul-port> \
    jobs/gloo.nomad

If running locally or with vagrant, you can omit the address flags from the deployment command:

levant deploy \
    -var-file variables/variables-linux.yaml \
    jobs/gloo.nomad

You can monitor the status of the deployment job by executing the following command:

nomad job status gloo

When the deployment is complete, you are ready to deploy the Pet Store application to demonstrate Gloo’s capabilities.

Installing Gloo on Nomad (Mac)

In this step we will deploy Gloo using Levant on a macOS-based system. The assumption is that you are running Consul, Nomad, and Vault locally.

levant deploy \
    -var-file variables/variables-mac.yaml \
    jobs/gloo.nomad

You can monitor the status of the deployment job by executing the following command:

nomad job status gloo

When the deployment is complete, you are ready to deploy the Pet Store application to demonstrate Gloo’s capabilities.


Deploying a Sample Application

In this step we will deploy a sample application to demonstrate the capabilities of the Gloo Gateway on either your Linux or macOS system. We’re going to deploy the Pet Store application to Nomad using Levant.

Deploy the Pet Store on Nomad (Linux)

We will deploy the Pet Store application using Levant and Nomad on your local or remote Linux machine.

If you are running these services remotely, then you will need to update the address and consul-address values with your configuration. The default port for Nomad is 4646 and for Consul is 8500. Make sure to give the full address to your Nomad and Consul servers, e.g. https://my.consul.local:8500.

levant deploy \
    -var-file variables/variables-linux.yaml \
    -address <nomad-host>:<nomad-port> \
    -consul-address <consul-host>:<consul-port> \
    jobs/petstore.nomad

If running locally or with vagrant, you can omit the address flags from the deployment command:

levant deploy \
    -var-file variables/variables-linux.yaml \
    jobs/petstore.nomad

You can monitor the status of the deployment job by executing the following command:

nomad job status petstore

When the deployment is complete, you are ready to create a route for the Pet Store application.

Deploy the Pet Store on Nomad (Mac)

We will deploy the Pet Store application using Levant and Nomad on your local macOS machine.

levant deploy \
    -var-file variables/variables-mac.yaml \
    jobs/petstore.nomad

You can monitor the status of the deployment job by executing the following command:

nomad job status petstore

When the deployment is complete, you are ready to create a route for the Pet Store application.

Create a Route to the PetStore

We can now use glooctl to create a route to the Pet Store app we just deployed:

glooctl add route \
    --path-prefix / \
    --dest-name petstore \
    --prefix-rewrite /api/pets \
    --use-consul
{"level":"info","ts":"2019-08-22T17:15:24.117-0400","caller":"selectionutils/virtual_service.go:100","msg":"Created new default virtual service","virtualService":"virtual_host:<domains:\"*\" > status:<> metadata:<name:\"default\" namespace:\"gloo-system\" > "}
+-----------------+--------------+---------+------+---------+-----------------+--------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  | STATUS  | LISTENERPLUGINS |             ROUTES             |
+-----------------+--------------+---------+------+---------+-----------------+--------------------------------+
| default         |              | *       | none | Pending |                 | / -> gloo-system.petstore      |
|                 |              |         |      |         |                 | (upstream)                     |
+-----------------+--------------+---------+------+---------+-----------------+--------------------------------+

The --use-consul flag tells glooctl to write configuration to Consul Key-Value storage

The route will send traffic from the root of the Gloo Gateway to the prefix /api/pets on the Pet Store application. You can test that by using curl against the Gateway Proxy URL:

curl <nomad-host>:8080/

If running on macOS or with Vagrant:

curl localhost:8080/

If running on Linux, use the Host IP on the docker0 interface:

curl 172.17.0.1:8080/

Curl will return the following JSON payload from the Pet Store application.

[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]

Next Steps

Congratulations! You’ve successfully deployed Gloo to Nomad and created your first route. Now let’s delve deeper into the world of Gloo routing.

Most of the existing tutorials for Gloo use Kubernetes as the underlying resource, but they can also use Nomad. Remember that all glooctl commands should be used with the --use-consul flag, and deployments will need to be orchestrated through Nomad instead of Kubernetes.