Previously, in Part 1 I described Kubernetes Controllers and the Operator pattern. In Part 2, I explained why Controllers are important and how Metacontroller makes it easier to build them in your favorite language.
In this, the 3rd and final part of this series, I’ll show off Metatron, which is my Ruby-based approach to building Metacontrollers. We’ll walk through building a basic CompositeController
responsible for a custom Blog
Kubernetes resource. I’m still working on the gem, but this should be a good demonstration of its capabilities and how easy it is to get started. Much of this post is a recap from Part 2, but it seems worthwhile to present it all together.
Our Requirements
First, let’s go over what we’ll be building. The goal is to be able to create blog sites, but without having to go through all the work of creating all the required Kubernetes resources ourselves. We still want the power of Kubernetes, but without all the complexity. Ideally, this means we’ll just make a single Blog
resource with a minimal spec, and our Controller will take care of the rest for us.
We need the Controller to create:
- A database we can connect to for persistence
- This can be a
StatefulSet
- We’ll also need a
Service
to allow connecting - Plus a
Secret
for storing user credentials
- This can be a
- A
Secret
that contains the database connection details (for use by the blog application) - A
Deployment
for the application itself- We’ll also make a
Service
for the app’s HTTP port - Plus we’ll make an
Ingress
for exposing the app outside the cluster
- We’ll also make a
That’s a fair amount of Kubernetes resources we’ll create. Let’s think through what these resource will need to create successfully.
For the DB, we can make some assumptions just to get started. We’ll hard-code the Docker image it runs to be mysql:8.0
and we’ll be boring about how we create credentials. We can also derive the name of the DB from the blog name.
For the app, I’m going to go with wordpress:6
, since I know how to get it up and running quickly, but we’ll keep that configurable. It listens on port 80 and only needs a few environment variables and one volume.
To keep things simple, let’s provide a single offering to our users:
- 20GB of disk space (5GB for the DB and 15GB for WordPress)
- 1 domain name with free TLS (thanks to Let’s Encrypt)
- 2 app replicas for HA
We’ll make all these configurable though, in case we want to change them later. This means that we’ll want our Custom Resource to look something like this:
1 2 3 4 5 6 7 8 9 10 |
apiVersion: therubyist.org/v1 kind: Blog metadata: name: test spec: image: wordpress:6 replicas: 2 storage: app: 15Gi db: 5Gi |
With that, I think we know what we’re hoping to achieve and why.
Laying the Groundwork
Before we hack on any Ruby code (or any of its supporting components), we’ll need to inform Kubernetes about this new, custom Blog
resource. We can create it via the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: blogs.therubyist.org spec: group: therubyist.org names: kind: Blog plural: blogs singular: blog scope: Namespaced versions: - name: v1 served: true storage: true subresources: status: {} schema: openAPIV3Schema: type: object properties: spec: type: object properties: image: type: string replicas: type: integer minimum: 1 storage: type: object properties: app: type: string db: type: string |
That might look like a lot, but it is pretty easy to follow. It is essentially just an OpenAPI3 schema for the custom resource we described above.
Apply that to your Kubernetes cluster:
1 |
kubectl apply -f blog-crd.yaml |
Now we need to create a CompositeController
resource that instructs Metacontroller to watch for our new custom resources. This assumes that Metacontroller is already installed. It’s pretty easy to use kustomize to install it, but there are other methods as well.
We can use a slightly modified version of the CompositeController
resource from part 2:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
apiVersion: metacontroller.k8s.io/v1alpha1 kind: CompositeController metadata: name: blog-controller spec: generateSelector: true parentResource: apiVersion: therubyist.org/v1 resource: blogs childResources: - apiVersion: apps/v1 resource: deployments updateStrategy: method: InPlace - apiVersion: apps/v1 resource: statefulsets updateStrategy: method: InPlace - apiVersion: v1 resource: services updateStrategy: method: InPlace - apiVersion: networking.k8s.io/v1 resource: ingresses updateStrategy: method: InPlace - apiVersion: v1 resource: secrets updateStrategy: method: InPlace hooks: sync: webhook: service: name: blog-controller namespace: blog-controller port: 9292 protocol: http path: /sync |
Run this to apply the resource:
1 |
kubectl apply -f blog-controller.yaml |
This instructs Metacontroller that, when it encounters changes to blogs
, it should hit the /sync
endpoint on the blog-controller
service in the blog-controller
namespace on port 9292
. Metacontroller will provide a list of child resources for any encountered Blog
and will expect a response that describes the new desired state of these (or other) resources.
Creating a Metatron Controller
With all that boilerplate Metacontroller stuff out of the way, we can get to work on the Metatron controller. Much of this will be a repeat of what is in the Metatron repo’s README.
A Typical App Skeleton
As with most Ruby/Sinatra-based projects, we’ll need to prepare a basic starting point for building and deploying a Ruby application. This includes things like a Gemfile
, a config.ru
, and a Dockerfile
. If you’re unfamiliar with these, there are tons of great resources that can help explain what they’re doing (plus I add comments to make it clear). Leave a comment if you get stuck.
First, let’s make a directory to get started on our development:
1 |
git init blog_controller && cd blog_controller |
This made a new directory and started us out with a new git
repo to track our code changes.
From there, let’s create the Gemfile
; this is required to make sure we’re installing the dependencies required to run our Metatron controller:
1 2 3 4 5 |
# frozen_string_literal: true source "https://rubygems.org" gem "metatron" |
Yup, that’s really all that we’ll need. Let’s install things:
1 |
bundle install |
You should now have a Gemfile.lock
file which pins installed libraries to specific versions. The lock file makes sure your development environment is the same as what you deploy to production.
Now, let’s create our config.ru
file. This informs rack
about how to route incoming requests to our code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# frozen_string_literal: true # \ -s puma require "metatron" # Loads our custom code require_relative "./lib/blog_controller/sync" use Rack::ShowExceptions use Rack::Deflater mappings = { # This one is built-in to Metatron and is useful for monitoring "/ping" => Metatron::Controllers::Ping.new, # We'll need to make this one "/sync" => BlogController::Sync.new, # You might add other controllers, like a finalizer or a customize hook } # Runs the rack application (the entire web app) run Rack::URLMap.new(mappings) |
Finally, we’ll need to make a Dockerfile
to build a container image with our code in it:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
FROM ruby:3.1 RUN mkdir -p /app COPY config.ru /app/ COPY Gemfile /app/ COPY Gemfile.lock /app/ COPY lib/ /app/lib/ RUN apt update && apt upgrade -y RUN useradd appuser -d /app -M -c "App User" RUN chown appuser /app/Gemfile.lock USER appuser WORKDIR /app RUN bundle install ENTRYPOINT ["bundle", "exec"] CMD ["puma"] |
A Metatron Controller
The code for our controller might start out with something like this (don’t worry, we’ll walk through some of it):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 |
# frozen_string_literal: true module BlogController class Sync < Metatron::SyncController # This method needs to return a Hash which will be converted to JSON # It should have the keys "status" (a Hash) and "children" (an Array) def sync # request_body is a convenient way to access the data provided by MetaController parent = request_body["parent"] existing_children = request_body["children"] desired_children = [] # first, let's create the DB and its service desired_children += construct_db_resources(parent, existing_children) # now let's make the app and its parts db_secret = desired_children.find { |r| r.kind == "Secret" && r.name.end_with?("db") } desired_children += construct_app_resources(parent, db_secret) # We might eventually want a mechanism to build status based on the world: # status = compare_children(request_body["children"], desired_children) status = {} { status:, children: desired_children.map(&:render) } end def construct_app_resources(parent, db_secret) resources = [] app_db_secret = construct_app_secret(parent["metadata"], db_secret) resources << app_db_secret app_deployment = construct_app_deployment( parent["metadata"], parent["spec"], app_db_secret ) resources << app_deployment app_service = construct_service(parent["metadata"], app_deployment) resources << app_service resources << construct_ingress(parent["metadata"], app_service) resources end def construct_db_resources(parent, existing_children) resources = [] db_secret = construct_db_secret(parent["metadata"], existing_children["Secret.v1"]) resources << db_secret db_stateful_set = construct_db_stateful_set(db_secret) resources << db_stateful_set db_service = construct_service( parent["metadata"], db_stateful_set, name: "db", port: 3306 ) resources << db_service resources end def construct_db_stateful_set(secret) stateful_set = Metatron::Templates::StatefulSet.new("db") stateful_set.image = "mysql:8.0" stateful_set.additional_pod_labels = { "app.kubernetes.io/component": "db" } stateful_set.envfrom << secret.name stateful_set end def construct_app_deployment(meta, spec, auth_secret) deployment = Metatron::Templates::Deployment.new(meta["name"], replicas: spec["replicas"]) deployment.image = spec["image"] deployment.additional_pod_labels = { "app.kubernetes.io/component": "app" } deployment.env["WORDPRESS_DB_HOST"] = "db" deployment.envfrom << auth_secret.name deployment end def construct_ingress(meta, service) ingress = Metatron::Templates::Ingress.new(meta["name"]) ingress.add_rule( "#{meta["name"]}.blogs.therubyist.org": { service.name => service.ports.first[:name] } ) ingress.add_tls("#{meta["name"]}.blogs.therubyist.org") ingress end def construct_service(meta, resource, name: meta["name"], port: "80") service = Metatron::Templates::Service.new(name, port) service.additional_selector_labels = resource.additional_pod_labels service end def construct_app_secret(meta, db_secret) # We'll want to use the password we specified for the DB user user_pass = db_secret.data["MYSQL_PASSWORD"] Metatron::Templates::Secret.new( "#{meta["name"]}app", { "WORDPRESS_DB_USER" => meta["name"], "WORDPRESS_DB_PASSWORD" => user_pass, "WORDPRESS_DB_NAME" => meta["name"] } ) end def construct_db_secret(meta, existing_secrets) name = "#{meta["name"]}db" existing = (existing_secrets || {})[name] data = if existing { "MYSQL_ROOT_PASSWORD" => Base64.decode64(existing.dig("data", "MYSQL_ROOT_PASSWORD")), "MYSQL_DATABASE" => Base64.decode64(existing.dig("data", "MYSQL_DATABASE")), "MYSQL_USER" => Base64.decode64(existing.dig("data", "MYSQL_USER")), "MYSQL_PASSWORD" => Base64.decode64(existing.dig("data", "MYSQL_PASSWORD")) } else { "MYSQL_ROOT_PASSWORD" => SecureRandom.urlsafe_base64(12), "MYSQL_DATABASE" => meta["name"], "MYSQL_USER" => meta["name"], "MYSQL_PASSWORD" => SecureRandom.urlsafe_base64(8) } end Metatron::Templates::Secret.new(name, data) end end end |
I know that looks like a lot of code, but there’s quite a bit going on in there. It generates over 250 lines of JSON (when prettified, at least).
Let’s walk through some of the important code. We’re creating a subclass of Metatron::SyncController
, which means this controller can handle /sync
requests. This defers to the sync
method which looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
def sync # request_body is a convenient way to access the data provided by MetaController parent = request_body["parent"] existing_children = request_body["children"] desired_children = [] # first, let's create the DB and its service desired_children += construct_db_resources(parent, existing_children) # now let's make the app and its parts db_secret = desired_children.find { |r| r.kind == "Secret" && r.name.end_with?("db") } desired_children += construct_app_resources(parent, db_secret) # We might eventually want a mechanism to build status based on the world: # status = compare_children(request_body["children"], desired_children) status = {} { status:, children: desired_children.map(&:render) } end |
Metatron offers some handy helper methods to make things easy. For instance request_body
is the parsed request body. There are others, but I have some documentation work to do.
Our goal is to construct a list of desired children and provide them in the response body under the key children
. To construct this list of children, we call some other methods we created to provide Template
instances based on Metatron-provided templates. Calling render
on a Template
causes it to provide a ruby Hash that can be converted to JSON.
Put another way, as long as sync
provides a top-level Hash that has the keys status
and children
, where status
is a Hash and children
is an Array of Hashes, things will work. You don’t need to use Metatron templates; just provide a Hash representation of any Kubernetes resource and it’ll work.
To build the DB-related resources for our blogs, we use construct_db_resources
, which looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 |
def construct_db_resources(parent, existing_children) resources = [] db_secret = construct_db_secret(parent["metadata"], existing_children["Secret.v1"]) resources << db_secret db_stateful_set = construct_db_stateful_set(db_secret) resources << db_stateful_set db_service = construct_service( parent["metadata"], db_stateful_set, name: "db", port: 3306 ) resources << db_service resources end |
We’re building a Secret
, a StatefulSet
, and a Service
, each via their own method. Check out those methods to see how to do that via Metatron templates.
For the app-related resources, we use construct_app_resources, which looks like:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
def construct_app_resources(parent, db_secret) resources = [] app_db_secret = construct_app_secret(parent["metadata"], db_secret) resources << app_db_secret app_deployment = construct_app_deployment( parent["metadata"], parent["spec"], app_db_secret ) resources << app_deployment app_service = construct_service(parent["metadata"], app_deployment) resources << app_service resources << construct_ingress(parent["metadata"], app_service) resources end |
This creates a Secret
, Deployment
, a Service, and an
Ingress. There’s also a relationship between some of the DB resources (like for our app Secret) which you can dive into if you’d like.
With these two helper methods, we’ve got quite a list of Kubernetes resources we’ll create! We just collect those up into desired_children
and then return those in sync
.
That’s all there is to writing our controller! If you have questions about the code, please leave a comment and I can help. There’s a lot of room to expand on that simple example to build some very powerful capabilities into your controller. For what it’s worth, the above code will actually produce a working WordPress site (though you will no doubt need to tweak the domain we generate for it).
Building Our Metatron Controller
Now that we’ve got our controller code written, we need to build and deploy it so Metacontroller can use it. This means we’ll need to build an image and deploy it to Kubernetes. Luckily, this is pretty easy.
Building an image is a matter of running the right command:
1 |
docker build -t blogcontroller:latest . |
Note that the tag there (blogcontroller:latest
) will need to be pushed somewhere that Kubernetes can pull it. This is likely a private registry.
Deploying the Controller
There’s nothing particularly special about deploying the controller. This should look like a typical Kubernetes Deployment
. First, make a namespace to run the controller and its Service
:
1 |
kubectl create namespace blog-controller |
Here’s an example of the Kubernetes resources that will get things up and running (be sure to adjust the container image to match wherever you pished the above image):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
apiVersion: apps/v1 kind: Deployment metadata: name: blog-controller spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app.kubernetes.io/component: blog-controller template: metadata: labels: app.kubernetes.io/name: blog-controller app.kubernetes.io/component: blog-controller spec: containers: - name: server image: blogcontroller:latest imagePullPolicy: Always readinessProbe: httpGet: path: /ping port: controller timeoutSeconds: 6 initialDelaySeconds: 2 periodSeconds: 5 failureThreshold: 3 ports: - name: controller containerPort: 9292 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: blog-controller labels: app.kubernetes.io/name: blog-controller app.kubernetes.io/component: blog-controller-service spec: type: ClusterIP ports: - port: 9292 targetPort: 9292 protocol: TCP name: controller selector: app.kubernetes.io/component: blog-controller |
Now we just need to apply the Kubernetes YAML:
1 |
kubectl -n blog-controller apply -f blog-controller-k8s.yaml |
This should create the Deployment
and Service
that Metacontroller is looking for.
Using the Controller
All that’s left is to make a new Blog
! Taking the original example at the beginning of this post, we just need a namespace to use it:
1 |
kubectl create namespace test-blog |
Now to create the blog:
1 |
kubectl -n test-blog apply -f test-blog.yaml |
This should create the Blog
and all the child resources. You can check on it with things like these commands:
1 2 3 4 |
$ kubectl -n test-blog describe Blog/test-blog $ kubectl -n test-blog get Deployments $ kubectl -n test-blog get StatefulSets $ kubectl -n test-blog get Services |
If things went well, you should be able to hit your new fancy site, all thanks to surprisingly little Ruby code.
One thought on “Kubernetes Controllers via Metatron (Part 3)”
Comments are closed.