The Rubyist Blog

My musings on technology, science, math, and more

Kubernetes Controllers via Metatron (Part 3)

Published in

•

Metatron's Cube

Previously, in Part 1 I described Kubernetes Controllers and the Operator pattern. In Part 2, I explained why Controllers are important and how Metacontroller makes it easier to build them in your favorite language.

In this, the 3rd and final part of this series, I’ll show off Metatron, which is my Ruby-based approach to building Metacontrollers. We’ll walk through building a basic CompositeController responsible for a custom Blog Kubernetes resource. I’m still working on the gem, but this should be a good demonstration of its capabilities and how easy it is to get started. Much of this post is a recap from Part 2, but it seems worthwhile to present it all together.

Our Requirements

First, let’s go over what we’ll be building. The goal is to be able to create blog sites, but without having to go through all the work of creating all the required Kubernetes resources ourselves. We still want the power of Kubernetes, but without all the complexity. Ideally, this means we’ll just make a single Blog resource with a minimal spec, and our Controller will take care of the rest for us.

We need the Controller to create:

  • A database we can connect to for persistence
    • This can be a StatefulSet
    • We’ll also need a Service to allow connecting
    • Plus a Secret for storing user credentials
  • A Secret that contains the database connection details (for use by the blog application)
  • A Deployment for the application itself
    • We’ll also make a Service for the app’s HTTP port
    • Plus we’ll make an Ingress for exposing the app outside the cluster

That’s a fair amount of Kubernetes resources we’ll create. Let’s think through what these resource will need to create successfully.

For the DB, we can make some assumptions just to get started. We’ll hard-code the Docker image it runs to be mysql:8.0 and we’ll be boring about how we create credentials. We can also derive the name of the DB from the blog name.

For the app, I’m going to go with wordpress:6, since I know how to get it up and running quickly, but we’ll keep that configurable. It listens on port 80 and only needs a few environment variables and one volume.

To keep things simple, let’s provide a single offering to our users:

  • 20GB of disk space (5GB for the DB and 15GB for WordPress)
  • 1 domain name with free TLS (thanks to Let’s Encrypt)
  • 2 app replicas for HA

We’ll make all these configurable though, in case we want to change them later. This means that we’ll want our Custom Resource to look something like this:

apiVersion: therubyist.org/v1
kind: Blog
metadata:
  name: test
spec:
  image: wordpress:6
  replicas: 2
  storage:
    app: 15Gi
    db: 5Gi

With that, I think we know what we’re hoping to achieve and why.

Laying the Groundwork

Before we hack on any Ruby code (or any of its supporting components), we’ll need to inform Kubernetes about this new, custom Blog resource. We can create it via the following:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: blogs.therubyist.org
spec:
  group: therubyist.org
  names:
    kind: Blog
    plural: blogs
    singular: blog
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    subresources:
      status: {}
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              image:
                type: string
              replicas:
                type: integer
                minimum: 1
              storage:
                type: object
                properties:
                  app:
                    type: string
                  db:
                    type: string

That might look like a lot, but it is pretty easy to follow. It is essentially just an OpenAPI3 schema for the custom resource we described above.

Apply that to your Kubernetes cluster:

kubectl apply -f blog-crd.yaml

Now we need to create a CompositeController resource that instructs Metacontroller to watch for our new custom resources. This assumes that Metacontroller is already installed. It’s pretty easy to use kustomize to install it, but there are other methods as well.

We can use a slightly modified version of the CompositeController resource from part 2:

apiVersion: metacontroller.k8s.io/v1alpha1
kind: CompositeController
metadata:
  name: blog-controller
spec:
  generateSelector: true
  parentResource:
    apiVersion: therubyist.org/v1
    resource: blogs
  childResources:
  - apiVersion: apps/v1
    resource: deployments
    updateStrategy:
      method: InPlace
  - apiVersion: apps/v1
    resource: statefulsets
    updateStrategy:
      method: InPlace
  - apiVersion: v1
    resource: services
    updateStrategy:
      method: InPlace
  - apiVersion: networking.k8s.io/v1
    resource: ingresses
    updateStrategy:
      method: InPlace
  - apiVersion: v1
    resource: secrets
    updateStrategy:
      method: InPlace
  hooks:
    sync:
      webhook:
        service:
          name: blog-controller
          namespace: blog-controller
          port: 9292
          protocol: http
        path: /sync

Run this to apply the resource:

kubectl apply -f blog-controller.yaml

This instructs Metacontroller that, when it encounters changes to blogs, it should hit the /sync endpoint on the blog-controller service in the blog-controller namespace on port 9292. Metacontroller will provide a list of child resources for any encountered Blog and will expect a response that describes the new desired state of these (or other) resources.

Creating a Metatron Controller

With all that boilerplate Metacontroller stuff out of the way, we can get to work on the Metatron controller. Much of this will be a repeat of what is in the Metatron repo’s README.

A Typical App Skeleton

As with most Ruby/Sinatra-based projects, we’ll need to prepare a basic starting point for building and deploying a Ruby application. This includes things like a Gemfile, a config.ru, and a Dockerfile. If you’re unfamiliar with these, there are tons of great resources that can help explain what they’re doing (plus I add comments to make it clear). Leave a comment if you get stuck.

First, let’s make a directory to get started on our development:

git init blog_controller && cd blog_controller

This made a new directory and started us out with a new git repo to track our code changes.

From there, let’s create the Gemfile; this is required to make sure we’re installing the dependencies required to run our Metatron controller:

# frozen_string_literal: true

source "https://rubygems.org"

gem "metatron"

Yup, that’s really all that we’ll need. Let’s install things:

bundle install

You should now have a Gemfile.lock file which pins installed libraries to specific versions. The lock file makes sure your development environment is the same as what you deploy to production.

Now, let’s create our config.ru file. This informs rack about how to route incoming requests to our code:

# frozen_string_literal: true

# \ -s puma

require "metatron"
# Loads our custom code
require_relative "./lib/blog_controller/sync"

use Rack::ShowExceptions
use Rack::Deflater

mappings = {
  # This one is built-in to Metatron and is useful for monitoring
  "/ping" => Metatron::Controllers::Ping.new,
  # We'll need to make this one
  "/sync" => BlogController::Sync.new,
  # You might add other controllers, like a finalizer or a customize hook
}

# Runs the rack application (the entire web app)
run Rack::URLMap.new(mappings)

Finally, we’ll need to make a Dockerfile to build a container image with our code in it:

FROM ruby:3.1

RUN mkdir -p /app

COPY config.ru /app/
COPY Gemfile /app/
COPY Gemfile.lock /app/
COPY lib/ /app/lib/

RUN apt update && apt upgrade -y
RUN useradd appuser -d /app -M -c "App User"
RUN chown appuser /app/Gemfile.lock

USER appuser
WORKDIR /app
RUN bundle install

ENTRYPOINT ["bundle", "exec"]
CMD ["puma"]

A Metatron Controller

The code for our controller might start out with something like this (don’t worry, we’ll walk through some of it):

# frozen_string_literal: true

module BlogController
  class Sync < Metatron::SyncController
    # This method needs to return a Hash which will be converted to JSON
    # It should have the keys "status" (a Hash) and "children" (an Array)
    def sync
      # request_body is a convenient way to access the data provided by MetaController
      parent = request_body["parent"]
      existing_children = request_body["children"]
      desired_children = []

      # first, let's create the DB and its service
      desired_children += construct_db_resources(parent, existing_children)

      # now let's make the app and its parts
      db_secret = desired_children.find { |r| r.kind == "Secret" && r.name.end_with?("db") }
      desired_children += construct_app_resources(parent, db_secret)

      # We might eventually want a mechanism to build status based on the world:
      #   status = compare_children(request_body["children"], desired_children)
      status = {}

      { status:, children: desired_children.map(&:render) }
    end

    def construct_app_resources(parent, db_secret)
      resources = []
      app_db_secret = construct_app_secret(parent["metadata"], db_secret)
      resources << app_db_secret
      app_deployment = construct_app_deployment(
        parent["metadata"], parent["spec"], app_db_secret
      )
      resources << app_deployment
      app_service = construct_service(parent["metadata"], app_deployment)
      resources << app_service
      resources << construct_ingress(parent["metadata"], app_service)
      resources
    end

    def construct_db_resources(parent, existing_children)
      resources = []
      db_secret = construct_db_secret(parent["metadata"], existing_children["Secret.v1"])
      resources << db_secret
      db_stateful_set = construct_db_stateful_set(db_secret)
      resources << db_stateful_set
      db_service = construct_service(
        parent["metadata"], db_stateful_set, name: "db", port: 3306
      )
      resources << db_service
      resources
    end

    def construct_db_stateful_set(secret)
      stateful_set = Metatron::Templates::StatefulSet.new("db")
      stateful_set.image = "mysql:8.0"
      stateful_set.additional_pod_labels = { "app.kubernetes.io/component": "db" }
      stateful_set.envfrom << secret.name
      stateful_set
    end

    def construct_app_deployment(meta, spec, auth_secret)
      deployment = Metatron::Templates::Deployment.new(meta["name"], replicas: spec["replicas"])
      deployment.image = spec["image"]
      deployment.additional_pod_labels = { "app.kubernetes.io/component": "app" }
      deployment.env["WORDPRESS_DB_HOST"] = "db"
      deployment.envfrom << auth_secret.name
      deployment
    end

    def construct_ingress(meta, service)
      ingress = Metatron::Templates::Ingress.new(meta["name"])
      ingress.add_rule(
        "#{meta["name"]}.blogs.therubyist.org": { service.name => service.ports.first[:name] }
      )
      ingress.add_tls("#{meta["name"]}.blogs.therubyist.org")
      ingress
    end

    def construct_service(meta, resource, name: meta["name"], port: "80")
      service = Metatron::Templates::Service.new(name, port)
      service.additional_selector_labels = resource.additional_pod_labels
      service
    end

    def construct_app_secret(meta, db_secret)
      # We'll want to use the password we specified for the DB user
      user_pass = db_secret.data["MYSQL_PASSWORD"]
      Metatron::Templates::Secret.new(
        "#{meta["name"]}app",
        {
          "WORDPRESS_DB_USER" => meta["name"],
          "WORDPRESS_DB_PASSWORD" => user_pass,
          "WORDPRESS_DB_NAME" => meta["name"]
        }
      )
    end

    def construct_db_secret(meta, existing_secrets)
      name = "#{meta["name"]}db"
      existing = (existing_secrets || {})[name]
      data = if existing
        {
          "MYSQL_ROOT_PASSWORD" => Base64.decode64(existing.dig("data", "MYSQL_ROOT_PASSWORD")),
          "MYSQL_DATABASE" => Base64.decode64(existing.dig("data", "MYSQL_DATABASE")),
          "MYSQL_USER" => Base64.decode64(existing.dig("data", "MYSQL_USER")),
          "MYSQL_PASSWORD" => Base64.decode64(existing.dig("data", "MYSQL_PASSWORD"))
        }
      else
        {
          "MYSQL_ROOT_PASSWORD" => SecureRandom.urlsafe_base64(12),
          "MYSQL_DATABASE" => meta["name"],
          "MYSQL_USER" => meta["name"],
          "MYSQL_PASSWORD" => SecureRandom.urlsafe_base64(8)
        }
      end
      Metatron::Templates::Secret.new(name, data)
    end
  end
end

I know that looks like a lot of code, but there’s quite a bit going on in there. It generates over 250 lines of JSON (when prettified, at least).

Let’s walk through some of the important code. We’re creating a subclass of Metatron::SyncController, which means this controller can handle /sync requests. This defers to the sync method which looks like this:

    def sync
      # request_body is a convenient way to access the data provided by MetaController
      parent = request_body["parent"]
      existing_children = request_body["children"]
      desired_children = []

      # first, let's create the DB and its service
      desired_children += construct_db_resources(parent, existing_children)

      # now let's make the app and its parts
      db_secret = desired_children.find { |r| r.kind == "Secret" && r.name.end_with?("db") }
      desired_children += construct_app_resources(parent, db_secret)

      # We might eventually want a mechanism to build status based on the world:
      #   status = compare_children(request_body["children"], desired_children)
      status = {}

      { status:, children: desired_children.map(&:render) }
    end

Metatron offers some handy helper methods to make things easy. For instance request_body is the parsed request body. There are others, but I have some documentation work to do.

Our goal is to construct a list of desired children and provide them in the response body under the key children. To construct this list of children, we call some other methods we created to provide Template instances based on Metatron-provided templates. Calling render on a Template causes it to provide a ruby Hash that can be converted to JSON.

Put another way, as long as sync provides a top-level Hash that has the keys status and children, where status is a Hash and children is an Array of Hashes, things will work. You don’t need to use Metatron templates; just provide a Hash representation of any Kubernetes resource and it’ll work.

To build the DB-related resources for our blogs, we use construct_db_resources, which looks like this:

    def construct_db_resources(parent, existing_children)
      resources = []
      db_secret = construct_db_secret(parent["metadata"], existing_children["Secret.v1"])
      resources << db_secret
      db_stateful_set = construct_db_stateful_set(db_secret)
      resources << db_stateful_set
      db_service = construct_service(
        parent["metadata"], db_stateful_set, name: "db", port: 3306
      )
      resources << db_service
      resources
    end

We’re building a Secret, a StatefulSet, and a Service, each via their own method. Check out those methods to see how to do that via Metatron templates.

For the app-related resources, we use `construct_app_resources, which looks like:

    def construct_app_resources(parent, db_secret)
      resources = []
      app_db_secret = construct_app_secret(parent["metadata"], db_secret)
      resources << app_db_secret
      app_deployment = construct_app_deployment(
        parent["metadata"], parent["spec"], app_db_secret
      )
      resources << app_deployment
      app_service = construct_service(parent["metadata"], app_deployment)
      resources << app_service
      resources << construct_ingress(parent["metadata"], app_service)
      resources
    end

This creates a Secret, Deployment, a Service, and anIngress`. There’s also a relationship between some of the DB resources (like for our app Secret) which you can dive into if you’d like.

With these two helper methods, we’ve got quite a list of Kubernetes resources we’ll create! We just collect those up into desired_children and then return those in sync.

That’s all there is to writing our controller! If you have questions about the code, please leave a comment and I can help. There’s a lot of room to expand on that simple example to build some very powerful capabilities into your controller. For what it’s worth, the above code will actually produce a working WordPress site (though you will no doubt need to tweak the domain we generate for it).

Building Our Metatron Controller

Now that we’ve got our controller code written, we need to build and deploy it so Metacontroller can use it. This means we’ll need to build an image and deploy it to Kubernetes. Luckily, this is pretty easy.

Building an image is a matter of running the right command:

docker build -t blogcontroller:latest .

Note that the tag there (blogcontroller:latest) will need to be pushed somewhere that Kubernetes can pull it. This is likely a private registry.

Deploying the Controller

There’s nothing particularly special about deploying the controller. This should look like a typical Kubernetes Deployment. First, make a namespace to run the controller and its Service:

kubectl create namespace blog-controller

Here’s an example of the Kubernetes resources that will get things up and running (be sure to adjust the container image to match wherever you pished the above image):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog-controller
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/component: blog-controller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: blog-controller
        app.kubernetes.io/component: blog-controller
    spec:
      containers:
      - name: server
        image: blogcontroller:latest
        imagePullPolicy: Always
        readinessProbe:
          httpGet:
            path: /ping
            port: controller
          timeoutSeconds: 6
          initialDelaySeconds: 2
          periodSeconds: 5
          failureThreshold: 3
        ports:
        - name: controller
          containerPort: 9292
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: blog-controller
  labels:
    app.kubernetes.io/name: blog-controller
    app.kubernetes.io/component: blog-controller-service
spec:
  type: ClusterIP
  ports:
  - port: 9292
    targetPort: 9292
    protocol: TCP
    name: controller
  selector:
    app.kubernetes.io/component: blog-controller

Now we just need to apply the Kubernetes YAML:

kubectl -n blog-controller apply -f blog-controller-k8s.yaml

This should create the Deployment and Service that Metacontroller is looking for.

Using the Controller

All that’s left is to make a new Blog! Taking the original example at the beginning of this post, we just need a namespace to use it:

kubectl create namespace test-blog

Now to create the blog:

kubectl -n test-blog apply -f test-blog.yaml

This should create the Blog and all the child resources. You can check on it with things like these commands:

$ kubectl -n test-blog describe Blog/test-blog
$ kubectl -n test-blog get Deployments
$ kubectl -n test-blog get StatefulSets
$ kubectl -n test-blog get Services

If things went well, you should be able to hit your new fancy site, all thanks to surprisingly little Ruby code.

wordpress screenshow

Spread the love

One response