David - Musings of an SRE

Running Deis on Digitalocean - A Nginx/Haproxy LB Setup

DigitalOcean users generally tend to get the short end of the stick when it comes to convenience. When building your Deis cluster on DigitalOcean, you’ll realize you don’t get the benefit of being able to provision cloud-provider specific load balancers like (AWS ELB, GCE Load Balancers) to route inbound traffic into the kubernetes cluster and into your deis-router.

In this article, I’ll quickly go through setting up a (nginx + haproxy) load balancer that sits infront of deis.

user browser --> nginx (listening at :80) --> haproxy --> deis router NodePorts

Technically, nginx is optional if you have an haproxy that is able to listen at port 80. But if you’re like me and want to reuse an existing droplet that already has nginx listening at port 80, this would save you the trouble of setting up another additional droplet.

To elaborate:

Your user browser will hit your nginx that is listening at port 80. It will then listen to all incoming connections that doesn’t have a virtualhost that its expecting and forward them to a haproxy instance that forwards to all your nodes’ port 80 node ports.

Got it?

Ok, lets go.

Prerequisite

  • Deis Workflow is already setup

Deis Router

Get your IP address:

$ kubectl describe service deis-router --namespace=deis

Port:                   http    80/TCP
NodePort:               http    32478/TCP
Endpoints:              10.40.0.14:8080
Port:                   https   443/TCP
NodePort:               https   30673/TCP
Endpoints:              10.40.0.14:6443
Port:                   builder 2222/TCP
NodePort:               builder 31555/TCP
Endpoints:              10.40.0.14:2222
Port:                   healthz 9090/TCP
NodePort:               healthz 32379/TCP
Endpoints:              10.40.0.14:9090

Take note of all your Node Port and subsequent Ports it is referring to.

Nginx

In your nginx, add a new server field that will listen for all non-routed virtualhosts at port 80 and forward them out.

This will look like:

server {
  listen 80 default_server;

  location / {
    proxy_pass http://path/to/haproxy:port;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
  }
}

proxy_set_header Host here is really important as without it, all connections to Deis’ router will look like its just coming from localhost:8080. Your router will not know where to send the connection to as it is unable to determine which app you’re calling to. So remember, Add this!

Haproxy

A quick way to setup and configure your haproxy is to use an existing official docker image.

First, in whatever machine that you’re planning to run haproxy on, create a new directory where you want your haproxy config to live.

$ mkdir haproxy
$ cd haproxy

Now, lets create a bash script that we can start your haproxy

$ vim run.sh

# run.sh

#!/bin/bash
docker run -d -p 8080:8080 --name my-running-haproxy -v $(pwd):/usr/local/etc/haproxy/:ro haproxy:1.5

$ chmod +x run.sh

In the same directory, create a new file called haproxy.cfg

frontend localnodes
    bind *:8080
    mode http
    default_backend web

backend web
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server web01 <minion_node_1_ip>:32478
    server web02 <minion_node_2_ip>:32478

listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth someuser:password

What this does is to have your haproxy to listen at port 8080 and then forward to your minion node @ its node_ports. So any connection coming in at port 8080 gets forwarded to your deis-router.

Remember to also add in the code to listen for the other ports namely 443,2222 and 9090 and forward them to the respective NodePort as referenced in the Deis-Router section above

Note: If you’re not planning to run nginx and can have your haproxy listen at port 80, you can configure your haproxy.cfg to bind *:80 and just point your wildcard domains to your haproxy external ip.

Once that is done, lets run.

$ ./run.sh

Do a docker ps and you should see your haproxy running. If not, check your logs!

DNS

With that, head over to your DNS provider and create a wildcard domain that points to the haproxy.

For example to have <app>.tt.2launch.us, you need to create a A record on *.tt that points to the IP of your haproxy.

Then you can reach it when you make a call to:

$ deis register http://deis.tt.2launch.us

References