work, technical, docker, kubernetes, haproxy, varnish
10:10 -0400

At work, we recently switched from Tutum (now Docker Cloud) to Kubernetes. Part of that work was building up a load balancer. Kubernetes has built-in load balancing capabilities, but it only works with Google Compute or AWS, which we are not using. It also requires a public IP address for each service, which usually means extra (unnecessary) costs.

Having previously worked with Tutum's HAProxy image, I figured I could do the same thing with Kubernetes. A quick web search didn't turn up any existing project, so I quickly wrote my own. Basically, we have HAProxy handling all incoming HTTP(S) connections and passing them off to different services based on the Host header. There's a watcher that watches Kubernetes such as new/deleted pods for relevant changes and updates the HAProxy configuration so that it always sends requests to the right place. I also improved the setup by adding in a Varnish cache for some of our services. Here's how it all works.

We have two sets of pods: an set of HAProxy pods and a set of Varnish pods. Each pod has a Python process that watches etcd for Kubernetes changes, updates the appropriate (HAProxy or Varnish) configuration, and tells HAProxy/Varnish about the new configuration. Why do we watch etcd instead of using the Kubernetes API directly? Because, as far as I can tell, in the Kubernetes API, you can only watch one type of object (either pods, configmaps, secrets, etc.) for changes, whereas we need to watch multiple types at once, so dealing with the Kubernetes API means that we would need to make multiple simultaneous API requests, which would just make things more complicated.

Unlike Tutum's HAProxy image, which only allows you to change certain settings using environment variables, our entire configuration template is configurable using Jinja2 templates. This gives us a lot more flexibility, including being able to plug in Varnish fairly easily without having to make any code changes to the HAProxy configurator. Also, configuration variables for services are stored in their own ConfigMap, rather than as environment variables in the target pods which allows us to make configuration changes without restarting the pods.

When combining HAProxy and Varnish, one question to ask is how to arrange them: HAProxy in front of Varnish, or Varnish in front of HAProxy? We are using a setup similar to the one recommended in the HAProxy blog. In that setup, HAProxy handles all requests and passes non-cacheable requests directly to the backend servers. Cacheable requests are, of course, passed to Varnish. If Varnish has a cache miss, then it passes the request back to HAProxy, which then hands off the request to the backend server. As the article points out, in the event of a cache miss, there's a lot of requests, but cache misses should be very infrequent since Varnish only sees cacheable content. One main difference between the setup we have and the one in the article is that in the article, HAProxy listens on two IP addresses: one for requests coming from the public, and one for requests coming from Varnish. In our setup, we don't have two IP addresses for HAProxy to use. Instead, Varnish adds a request header that indicates that the request is coming from it, and HAProxy checks for that header.

At first, I set the Python process as the pod's command (the pod's PID 1), but ran into a slight issue. HAProxy reloads its configuration by, well, not reloading its configuration; it starts a new set of processes with the new configuration, which means that we ended up with a lot of zombie processes. To fix this, I could have changed the Python process to reap the zombies, but it was easier to just use Yelp's dumb-init instead.

We have the HAProxy pods managed as a DaemonSet, so it's running on every node, and the pods are set to use host networking for better performance. HAProxy itself is small enough that, at least with our current traffic, it doesn't affect the nodes much, so it isn't a problem for us right now to run it on every node. If we get enough traffic that it does make a difference, we can dedicate a node to it without much problem. One thing about this setup is that, even though it uses Kubernetes, HAProxy and Varnish don't need to be managed by Kubernetes. It just needs to be able to talk to etcd. So if we ever need a dedicated load balancer, we can spin up a node (or nodes) that just runs HAProxy and/or Varnish, say, using a DaemonSet and nodeSelector. Varnish is managed as a normal Kubernetes deployment and uses the normal container networking, so there's a bit of overhead there, but is fine for now. Again, if we have more concerns about performance, we can change our configuration easily enough.

It all seems to be working fairly well so far. There are some configuration tweaks that I'll have to go make, and there's one strange issue where Varnish doesn't like one of our services and just returns an empty response. But other than that, Varnish and HAProxy are just doing what they're supposed to do.

All the code is available on GitHub (HAProxy, Varnish).

0 Comments

technical, work, docker, kubernetes
09:42 -0400

Note: this is not a comprehensive comparison of Kubernetes and Docker Cloud. It is just based on my own experiences. I am also using Tutum and Docker Cloud more or less interchangeably, since Tutum became Docker Cloud.

At work, we used to use Tutum for orchestrating our Docker containers for ourCalculus practice problems site. While it was in beta, Tutum was free, but Tutum has now becomeDocker cloud and costs about $15 per month per managed node per month, on top of server costs. Although we got three free nodes since we were Tutum beta testers, we still felt the pricing was a bit steep, since the management costs would be more than the hosting costs. Even more so since we would have needed more private Docker repositories than what would have been included.

So I started looking for self-hosted alternatives. The one I settle on wasKubernetes, which originated from Google. Obviously, if you go self-hosted, you need to have enough system administration knowledge to do it, whereas with Docker Cloud, you don't need to know anything about system administration. It's also a bit more time consuming to set up — it took me about a week to set up Kubernetes (though most of that time was scripting the process so that we could do it again more quickly next time), whereas with Tutum, it took less than a day to get up and running.

Kubernetes will require at least one server for itself — if you want to ensure high availability, you'll want to run multiple masters. We're running on top of CoreOS, and a 512MB node seems a bit tight for the master for our setup. A 1GB node was big enough that, although they recommend not to, I allowed the master to schedule running other pods.

Kubernetes seems to have a large-ish overhead on the worker nodes (a.k.a. minions). Running top, the system processes take up at least 200MB, which means that on a 512MB node, you'd only have about 300MB to run your own pods unless you have swap space. I have no idea what the overhead on a Tutum/Docker cloud node was, since I didn't have access to check.

Previously, under Tutum, we were running on 5*512MB nodes, each of which had 512MB swap space. Currently, we're running on 3*1GB worker nodes plus 1*1GB master node (which also serves as a worker), no swap. (We'll probably need to add another worker in the near future (or maybe another combined master/worker) though under Tutum, we would have probably needed another node with the changes that I'm planning anyways.) Since we also moved from DigitalOcean toDreamHost (affiliate link) and their new DreamCompute service (which just came out of Beta as we were looking into self-hosting), our new setup ended up costing $1 less per month.

Under Tutum, the only way to pass in configuration (other than baking it into your Docker image, or unless you run your own configuration server) is through environment variables. With Kubernetes, you have more options, such as ConfigMaps and Secrets. That gives you more flexibility and allows (depending on your setup) on changing configuration on-the-fly. For example, I created anauto-updating HAProxy configurationthat allows you to specify a configuration template via a ConfigMap. When you update the ConfigMap, HAProxy gets immediately reconfigured with almost no downtime. This is in contrast to theTutum equivalent, in which a configuration change (via environment variables) would require a restart and hence more downtime.

The other configuration methods also allows the configuration to be more decoupled. For example, with Tutum's HAProxy, the configuration for a service such as virtual host names are specified using the target container's environment variables, which means that if you want to change the set of virtual hosts or the SSL certificate, you would need to restart your application containers. Since our application server takes a little while to restart, we want to avoid having to do that. On the other hand, if the configuration were set in HAProxy's environment, then it would be lost to other services that might want to use it (such as monitoring software that would might use the HTTP_CHECK variable). With a ConfigMap, however, the configuration does not need to belong to one side or the other; it can stand on its own, and so it doesn't interfere with the application container, and can be accessed by other pods.

Kubernetes can be all configured using YAML (or JSON) files, which means that everything can be version controlled. Under Tutum, things are primarily configured via the web interface, though they do have a command-line tool that you could use as well. However, the command-line tool uses a different syntax for creating versus updating, whereas with Kubernetes, you can just "kubectl apply -f", so even if you use the Tutum CLI and keep a script under version control for creating your services, it's easier to forget to change your script after you've changed a service.

There are a few things that Tutum does that Kubernetes doesn't do. For example, Tutum has built-in node management (if you use AWS, DigitalOcean, or one of the other providers that it is made to work with), whereas with Kubernetes, you're responsible for setting up your own nodes. Though there are apparently tools built on top of Kubernetes that do similar things, I never really looked into them, since we currently don't need to bring up/take down nodes very frequently. Tutum also has more deployment strategies (such as "emptiest node" and "high availability"), which was not that important for us, but might be more important for others.

Based on my experience so far, Kubernetes seems to be a better fit for us. For people who are unable/unwilling to administer their own servers, Docker Cloud would definitely be the better choice, and starting with Tutum definitely gave me time to look around in the Docker ecosystem before diving into a self-hosted solution.

0 Comments