I recently ran into an interesting issue with my home Kubernetes environment that runs my blog. As I mentioned in a previous post, I run my blog on k3s and I use cert-manager to manage my SSL certificates provided by Let’s Encrypt. Let’s say that I’ve temporarily changed my Internet provider and along with it, my router. This router does not appear to support NAT Loopback. The cert-manager documentation acknowledges the issue but doesn’t provide much of a solution. Cert-manager couldn’t renew my blog’s certificate because its self-check kept failing. I managed to solve the issue through a fairly simple CoreDNS change. Let’s take a look.
Category: Technology
How the Internet Works
One of my favorite questions to ask (or be asked) during an interview is the classic “how does the Internet work?” question. It usually goes something like this:
You open your favorite web browser, type in “www.mysite.com”, and hit return. Almost like magic, a fully-rendered web page shows up on your screen ready for you to view. Tell me, with as much detail as you can muster, what just happened.
The reason I like this question so much is that it isn’t just academic; it is a peek behind the curtains at what this person knows. It reveals what they’ve dug into in the past, learned in school, or dealt with while troubleshooting. The details show how much time they’ve spent demystifying the world (at least related to our working environment). Thinking about this made me realize how really fleshing out a solid answer to this would make a great blog post, so here we are.
Kubernetes Node Affinity and EBS Volumes
Occasionally, Kubernetes workloads require specialized nodes. Sometimes it’s for machine learning, or to save money through burstable node types, or maybe just to lock certain Pods to a dedicated subset of nodes. Thankfully, Kubernetes offers a few useful mechanisms to inform the scheduler how we’d like our workloads distributed: node-based or pod-based affinity rules along with taints and tolerations. I’ll go over how to use these briefly, but I use these frequently at work for numerous reasons. Recently, I realized something interesting about how Persistent Volume Claims (PVCs) work with dynamically provisioned storage like EBS volumes (meaning volumes that are created automatically by Kubernetes based on a StorageClass, rather than referencing existing volumes). The default behavior of a StorageClass is to immediately create a volume as soon as the PVC is created. This can have some consequences when trying to guide how Pods are scheduled.
LDAP in Containers
Most of the time, connecting to LDAP is pretty straightforward and is just a matter of applying the right configuration to your application. Or maybe it isn’t even something you need to think about; it could be abstracted away behind an API call. This wasn’t always the case though. In several of my previous jobs, authentication wasn’t just a matter of submitting a username and password; I needed to setup and maintain the system that made that work, both for the server and its clients. Thankfully there was a ton of documentation and guides for making Linux work with LDAP. But what about LDAP in containers?
Times have changed and now we’re building containers, not really needing to worry about a lot of the details of Linux configuration. For the most part, we don’t need much from PAM (and even less from sssd) in containers. That said, sometimes you encounter software that just has to rely on your OS for authentication where LDAP sure comes in handy. Here I describe how to configure your Docker container to leverage LDAP via sssd for users and groups.
Changing Your Public IP on Home Internet
Sometimes, when you’re using a Linux server as your home router/Internet gateway, you need to change your public IP. I won’t go into the reason(s) why, because they don’t really matter. Maybe you accidentally exposed the proxy port (mostly just for your kids to protect their Internet access) directly to the Internet and ended up blacklisted by most things on the Internet, who knows? Best not to dwell on the hypothetical.
At first, it seems obvious: just release and renew with your DHCP client. A quick dhclient -r enp4s0 (or whatever your interface name is) seems like the solution. But ISPs are too smart for that. Maybe try turning off your cable modem and leaving it off for a few minutes? Nope. None of this works because of how ISPs (and really, most any DHCP server) handle DHCP leases: they’re tied to the MAC address of your network interface. This means that when your network interface’s MAC address is seen by their DHCP server, it’ll offer it the same IP. This makes sense for ISPs to do; they can tie an IP to a customer based on their physical device.
Redirecting Domains on a Traefik Ingress
I recently posted about my experience with k3s and how I’m now using it to run my blog. I also mentioned my blog’s new domain and how I’m keeping the old name working. That involved changing the Ingress resource for my blog, so I’ll show how I updated it to accept the old domains and automatically redirect to my preferred domain without needing to make WordPress itself do any redirecting.
Blogging on Kubernetes
It has been a while since I last posted, but between college, work, and kids, I’ve been pretty busy. That said, I recently attended KubeCon 2019 and saw a lot of interesting presentations. As a fan of Rancher, I gravitated toward a lot of their talks. One that really caught my attention was Darren Shepherd’s talk on k3s. I really liked what I saw; it made setting up Kubernetes really easy, lightened the dependency load for small clusters, but still is very much the right amount of “batteries included” like most things made by Rancher.
I decided to move my home server (which runs, among other things, this blog) to k3s. Here, I’ll walk through how I did it — at least specifically for running a WordPress blog — just to demonstrate how easy it is. Fair warning though, there is a lot of YAML ahead!
Read-only Docker Containers
There are lots of good reasons for and articles recommending running Docker containers read-only, but what I have a difficult time finding are descriptions of how to do this for many popular images. Some software needs to write to a few important and predictable locations. It surprises me how often image providers neglect to offer instructions or details required to run their image this way.
Even setting aside read-only containers, counting on writing to the writable layer just feels wrong. Per the documentation, for the writable layer, both read and write speeds are lower because of the copy-on-write/overlay process through the storage driver. In my experience, docker diff output means I haven’t taken the time to configure my volume declarations, either through tmpfs mounts, volumes, or bind mounts.
Encryption Redux
In yet another exciting move for my blog, I’ve switched SSL providers from my previous provider to Let’s Encrypt. I’ve done so using a set of Docker containers, which also helped me move this WordPress blog to Docker as well. Now my blog is faster, encrypted for free, and easier to backup and maintain. I’ll probably post a brief article soon about how I set this up in more detail (including some docker-compose.yml snippets) soon.
Distributing CLI Tools via Docker
Throughout my career, I’ve seen a couple recurring patterns related to the tools I write: I write a lot of small CLI tools and I like to share them with my coworkers (and whenever possible, the rest of the world).
This has led to several iterations of solving the problem How do I make this tool easy to run? since I don’t want to burden people with understanding the intricacies of all my tools’ dependencies. These tend to be Ruby, some number of gems, and possibly some other common unix utilities. The solutions I’ve come up with have included a lengthy README with detailed instructions, Bundler with Rake tasks to do all the heavy lifting for non-Ruby things, fpm, and even “curl bash piping” (yes, I’m horrible).
Recently I decided to use Docker to solve this problem, since I’m using it so much anyway. Using Docker has some huge benefits for sharing applications of all types: the dependencies list gets whittled down to just Docker, things work on more platforms, testing gets simpler, and it is the new hotness which makes people say “whoa” and that’s fun. That said, the downsides can be frustrating: working with files on your machine gets messy, more typing with the extra Docker-related preamble, things are less straightforward and clear, simple mistakes can lead to lots of images and containers to clean up, and the executable gets significantly larger (since the Docker image is a whole, albeit lightweight, OS userland to run the app). After weighing these pros and cons, I’ve found that telling a coworker to docker pull registry.url/my/app and run it with --help is so much more convenient than the alternatives.