I recently ran into an interesting issue with my home Kubernetes environment that runs my blog. As I mentioned in a previous post, I run my blog on k3s and I use cert-manager to manage my SSL certificates provided by Let’s Encrypt. Let’s say that I’ve temporarily changed my Internet provider and along with it, my router. This router does not appear to support NAT Loopback. The cert-manager documentation acknowledges the issue but doesn’t provide much of a solution. Cert-manager couldn’t renew my blog’s certificate because its self-check kept failing. I managed to solve the issue through a fairly simple CoreDNS change. Let’s take a look.
How the Internet Works
One of my favorite questions to ask (or be asked) during an interview is the classic “how does the Internet work?” question. It usually goes something like this:
You open your favorite web browser, type in “www.mysite.com”, and hit return. Almost like magic, a fully-rendered web page shows up on your screen ready for you to view. Tell me, with as much detail as you can muster, what just happened.
The reason I like this question so much is that it isn’t just academic; it is a peek behind the curtains at what this person knows. It reveals what they’ve dug into in the past, learned in school, or dealt with while troubleshooting. The details show how much time they’ve spent demystifying the world (at least related to our working environment). Thinking about this made me realize how really fleshing out a solid answer to this would make a great blog post, so here we are.
Kubernetes Node Affinity and EBS Volumes
Occasionally, Kubernetes workloads require specialized nodes. Sometimes it’s for machine learning, or to save money through burstable node types, or maybe just to lock certain Pods to a dedicated subset of nodes. Thankfully, Kubernetes offers a few useful mechanisms to inform the scheduler how we’d like our workloads distributed: node-based or pod-based affinity rules along with taints and tolerations. I’ll go over how to use these briefly, but I use these frequently at work for numerous reasons. Recently, I realized something interesting about how Persistent Volume Claims (PVCs) work with dynamically provisioned storage like EBS volumes (meaning volumes that are created automatically by Kubernetes based on a StorageClass, rather than referencing existing volumes). The default behavior of a StorageClass is to immediately create a volume as soon as the PVC is created. This can have some consequences when trying to guide how Pods are scheduled.
RSpec Testing for Ruby AWS Lambda Functions
Recently, I wrote an AWS Lambda function at work in Ruby but I didn’t have a handy tool for creating a project skeleton like bundle gem does. That means nothing bootstrapped my testing for me. While copy+pasting code into pry proved that my simple function worked, that wasn’t quite good enough. What I really wanted was the ability to use RSpec with my Lambda code. After a cursory search of the Internet for some examples, I was left disappointed with how little I found. So, I rolled up my sleeves and figured it out.
Continue reading RSpec Testing for Ruby AWS Lambda Functions
Meet Bullion, An ACMEv2 Certificate Authority
I’m a huge fan of Let’s Encrypt and what they’ve done to secure the Internet. They’ve made safe communication free and open. Through their ACME protocol (and subsequent ACMEv2 protocol), they have changed PKI and the way we look at automating certificate provisioning for good. All that said, Let’s Encrypt only really helps for public-facing services; for internal domains and services, I want a solution that doesn’t require creating public records. I also don’t want to reinvent all the tooling; I want to be able to use ACMEv2 (through things like cert-manager on Kubernetes or certbot for EC2). Enter Bullion, an unassuming Ruby ACMEv2 Certificate Authority built specifically for internal domains.
Continue reading Meet Bullion, An ACMEv2 Certificate Authority
Rationality and COVID-19
The world is pretty crazy right now. COVID-19 has caused the closure of countless businesses and slowed down in-person business for most others (though there is a glimmer of hope as small business and schools are looking at reopening in some respect). Many, many people are without a job or facing some kind of cutback. This isn’t just a US tragedy either; the economic impacts are being felt around the world.
So the question is, why? Why has the world essentially ground to a halt? People get sick all the time, right?
Continue reading Rationality and COVID-19
Connected: A Ruby Graph Search Library
A common problem posed in coding competitions and undergraduate computer science classes is implementing an algorithm for finding the shortest path through a series of interconnected nodes. It can be phrased in many ways, mostly because graph theory (which is the domain of this particular problem) applies to so many areas of life. Things like driving directions, routing packets in a network, shipping logistics, and so many NP problems are examples of the same problem. It turns out that finding the shortest/lowest cost path through a graph is difficult. While Edsger Dijkstra wrote his famous algorithm to solve this problem back in 1956, there aren’t many libraries to actually use it, at least not for Ruby and in a practical sense. Sure, there are plenty of academic demonstrations but I couldn’t find anything that I would want to depend on as a library (it turns out that rgl is pretty great). This inspired me to write connected, which can be used to search weighted, directed, or undirected graphs.
Rewriting Ruby’s #pack and #unpack Methods in Crystal
In Ruby, especially while writing low-level protocols, you may have encountered Array#pack() and/or String#unpack(). If you’ve ever experimented with them, they may seem mysterious; they take a cryptic string as a parameter (maybe something like C4 or w*) and seem to return gibberish (or, conversely, convert gibberish into something humans can understand). I’m not going to go into tons of detail, nor am I going to cover every format that these methods accept, but I’ll cover a few that I’ve used.
What led me to writing this post was actually some recent experimenting that I’ve been doing with Crystal. I’m trying my hand at writing a simple BER parser/encoder as a part of my journey to writing an LDAP library for Crystal. The lack of an LDAP library is honestly the biggest reason I haven’t used Crystal for more things. Since BER is a binary method of encoding, the Ruby LDAP BER code uses a ton of Array#pack() and String#unpack(). Unfortunately, Crystal doesn’t have analogous methods for its Array class or String struct so I’ve had to write my own.
Here, I’ll describe a few of the formats supported by #pack and write some compatible examples in Crystal.
Continue reading Rewriting Ruby’s #pack and #unpack Methods in Crystal
LDAP in Containers
Most of the time, connecting to LDAP is pretty straightforward and is just a matter of applying the right configuration to your application. Or maybe it isn’t even something you need to think about; it could be abstracted away behind an API call. This wasn’t always the case though. In several of my previous jobs, authentication wasn’t just a matter of submitting a username and password; I needed to setup and maintain the system that made that work, both for the server and its clients. Thankfully there was a ton of documentation and guides for making Linux work with LDAP. But what about LDAP in containers?
Times have changed and now we’re building containers, not really needing to worry about a lot of the details of Linux configuration. For the most part, we don’t need much from PAM (and even less from sssd) in containers. That said, sometimes you encounter software that just has to rely on your OS for authentication where LDAP sure comes in handy. Here I describe how to configure your Docker container to leverage LDAP via sssd for users and groups.
Working During a Pandemic
We’re in very strange times, that’s for sure. The entirety of humankind is trying to social engineer a defense against a microscopic threat, yet here I am wanting to talk about working and how well it’s going in spite of the pandemic.
When I was first inspired to write this brief post, I decided against it because it felt too much like bragging. After all, not being able to work is clearly on the minds of MILLIONS of people right now. But really, this isn’t about me; it is a testament to how working on a phenomenal team makes it possible to be busy and happy with my job no matter what the rest of the world is up to.
I’m glad to work with friends, to have tons of work to do, and to see my projects succeeding. As a person that already works from home, this whole situation hasn’t been much of a change for me, but it has been fantastic to see how well the rest of the team has been doing with it. It really matters how companies handle situations like these; we all gain some solid insight into what a company values and how much it can adapt. My group has certainly adapted and after a week or so, I think we’re back in the groove. We’re not slowing down — we’re picking up! I’m grateful that work is the last of my worries (which is precisely how it should be during a pandemic).
Some tips for companies:
- Let people work from home if they can!
- Make sure your employees aren’t worried about taking sick time or personal time to figure out family situations.
- Lead by example: if CEOs or leadership video conferences from a home desk just like everyone else, people are less nervous.
- Be transparent.
- Be understanding.
Tips for individuals:
- Develop a healthy morning routine.
- Take breaks and go for a walk outside.
- Establish rules for disruptions during work (when my door is closed, I’m in a meeting).
- Communicate often in the chat.
- Leave room between meetings for personal time like getting water, checking in on kids, grabbing a snack, etc.
Stay healthy, everyone!