Home

Building Kubernetes Controllers in Node.js

Colin J. Ihrig

A Kubernetes controller is a piece of software that continuously watches at least one type of Kubernetes resource type for changes. When a watched resource does change, the controller is responsible for making any necessary changes to the cluster in order to achieve the desired state. This process is known as reconciliation.

Controllers and custom resources are often used together to implement the powerful operator pattern. The operator pattern allows many Kubernetes workloads to be managed via automation by reacting to events in the same way that a human operator would. There are existing operators available for many common applications such as PostgreSQL, Grafana, and Prometheus.

Like most things in the Kubernetes ecosystem, controllers are most often written in Golang. However, that is not a requirement! Because Kubernetes components typically communicate with each other over a network, a Kubernetes client is all you really need in order to create controllers and other interesting types of extensions.

Managing Complexity and Configuration

In the previous paragraph, I said you only need a Kubernetes client to get started. That is a bit of an exaggeration. Kubernetes is a CNCF project. The CNCF acronym is short for Cloud Native Computing Foundation, but those C's could easily stand for Complexity, Configuration, or even Codegen. In Kubernetes, it is common for even small pieces of code to be accompanied by large YAML manifests and other supporting artifacts. Luckily, these files can often be managed by tooling.

Kubebuilder is a popular command line tool for scaffolding controllers, Custom Resource Definitions (CRDs), and all of the accompanying configuration that comes with them. I created Kubenode to serve a similar purpose, but with Node.js projects. We will be using the Kubenode CLI for scaffolding throughout this post, but most of the concepts from The Kubebuilder Book still apply.

Create a Project

The remainder of this post will step through the creation of a controller that performs basic math. Start by creating a directory for your new project and initializing a new Kubenode project. For simplicity, this post uses npx to run the kubenode CLI without needing a separate installation step.

$ mkdir -p /tmp/math-project
$ cd /tmp/math-project
$ npx kubenode init -p math-project -d math.io

After running these commands, the following resources should exist:

  • kubenode.json - A JSON file that Kubenode uses to persist data about the resources that it manages. Users do not typically need to work directly in this file.
  • package.json - The package manifest for the project. It contains several npm scripts for managing the project.
  • Dockerfile - Kubernetes orchestrates containers. This file is used to build a container image for the project so that it can be deployed to a cluster.
  • config/ - A directory containing YAML manifests used to create Kubernetes resources and manage RBAC.
  • lib/ - A directory containing application and test code. This will be discussed in more detail in the next section.

The Application Code

The contents of lib/index.js are shown below:

import { Manager } from '@kubenode/controller-runtime';
// @kubenode:scaffold:imports

// ATTENTION: YOU **SHOULD** EDIT THIS FILE!

const manager = new Manager();
// @kubenode:scaffold:manager
await manager.start();

The @kubenode/controller-runtime library provides a runtime API for working with controllers. It borrows many concepts from the Golang controller-runtime library. The Manager class is used to register and collectively manage various controllers, webhooks, and other resources. There are not currently any other resources in the project, so there isn't much for the Manager to do.

The comments starting with the string '@kubenode:scaffold' are annotations used by Kubenode when updating the code. You should not modify or remove these lines unless you are sure what you are doing.

Add an API

Next, we will create a SimpleMath CRD and controller. This resource type will accept two numbers and an operation (add, subtract, etc.) as inputs, and return the result as an output. In Kubernetes terminology, the inputs will be part of the resource's spec, and the output will be part of the status.

Scaffold the SimpleMath resource using the following command. This command specifies the group (-g), version (v1), and kind (-k) of the resource to create. The group, version, and kind (GVK) is the standard way that Kubernetes uses to uniquely identify resource types.

$ npx kubenode add api -g math.io -v v1 -k SimpleMath

This command makes a number of changes to the existing project. First, the existing lib/index.js file is updated to include the new resource type as shown below. Kubenode used the existing annotation comments to import the newly created controller, construct a new instance, and register it with the Manager.

import { Manager } from '@kubenode/controller-runtime';
import { SimpleMathReconciler } from './controller/math.io_v1/simplemath.js';
// @kubenode:scaffold:imports

// ATTENTION: YOU **SHOULD** EDIT THIS FILE!

const manager = new Manager();
(new SimpleMathReconciler()).setupWithManager(manager);
// @kubenode:scaffold:manager
await manager.start();

Write the Controller

As you may be able to tell from the previous code, the controller is located in controller/math.io_v1/simplemath.js. Open that file, and update it to match the following snippet. We are updating the code such that when a reconcile is triggered, we retrieve the new state of the resource, compute the result based on the inputs, and update the status if necessary.

import {
  k8s,  // This is a reference to the @kubernetes/client-node package.
  newControllerManagedBy,
  Reconciler,
} from '@kubenode/controller-runtime';
// Create a client for interacting with the Kubernetes API.
const kubeconfig = new k8s.KubeConfig();
kubeconfig.loadFromDefault();
const client = kubeconfig.makeApiClient(k8s.CustomObjectsApi);

export class SimpleMathReconciler extends Reconciler {
  async reconcile(ctx, req) {
    console.log(`SimpleMath name=${req.name}, namespace=${req.namespace}, reconcileID=${ctx.reconcileID}`);

    // Retrieve the SimpleMath resource that was modified.
    const query = {
      group: 'math.io',
      version: 'v1',
      plural: 'simplemaths',
      name: req.name,
      namespace: req.namespace,
      body: undefined,  // This will be used below.
    };
    const resource = await client.getNamespacedCustomObject(query);

    // Determine the result based on the current state of the resource.
    const { number1, number2, operation } = resource.spec;
    let result;

    switch (operation.toLowerCase()) {
      case 'add':
        result = number1 + number2;
        break;
      case 'subtract':
        result = number1 - number2;
        break;
      case 'multiply':
        result = number1 * number2;
        break;
      case 'divide':
        result = number1 / number2;
        break;
      default:
        console.log(`Unsupported operation: ${operation}`);
        return;
    }

    // Ensure that the 'status' object exists.
    resource.status ??= {};

    // Update the resource with the new status only if necessary to avoid
    // triggering additional reconciliations.
    if (resource.status.result !== result) {
      resource.status.result = result;
      query.body = resource;
      await client.replaceNamespacedCustomObjectStatus(query);
    }
  }

  setupWithManager(manager) {
    newControllerManagedBy(manager)
      .for('SimpleMath', 'math.io/v1')
      .complete(this);
  }
}

Update the Types

In the controller/math.io_v1/ directory, there is another file named simplemath_types.ts. This file contains type definitions for the SimpleMath resource type. These types will be used to generate manifests for our new resource. Locate the SimpleMathSpec type, and update it as shown below:

type SimpleMathSpec = {
  /**
   * @description number1 is the first number to operate on.
   */
  number1: number;

  /**
   * @description number2 is the second number to operate on.
   */
  number2: number;

  /**
   * @description operation is the operation to perform.
   */
  operation: string;
};

Next, locate the SimpleMathStatus type and update it as shown below:

type SimpleMathStatus = {
  /**
   * @description result is the result of the operation.
   */
  result: number;
};

Use the Kubenode command shown below to generate the necessary manifests. See config/crd/math.io_v1_simplemath.yaml for the generated CustomResourceDefinition for our new resource type. Luckily, you shouldn't need to work in this file directly.

$ npx kubenode codegen -g math.io -v v1 -k SimpleMath

Add a Webhook

When developing controllers, one aspect that can be frustrating is performing input validation. The reason is that controllers operate in an asynchronous fashion. When a resource is created or modified, control flow immediately returns to the user. The controller asynchronously observes the change and then performs reconciliation. If an error occurs during reconciliation, the controller has limited options for returning the error to the user. Generally, asynchronous reconciliation errors should be reported via the resource's status object.

Input validation is a different type of error though. When a bad input is received, it is usually preferable to reject the change and immediately raise the error to the user. Kubernetes supports this use case via validating webhooks. A validating webhook is an HTTP callback that is separate from the controller, and is invoked when a write operation occurs on a resource. If the webhook allows the operation, then the write works as expected. However, if the webhook rejects the operation, then the write does not occur, and an error is returned to the user immediately.

The following Kubenode command generates a validating webhook for our SimpleMath resource. In this command, the -g, -v, and -k flags are again used to specify the GVK of the webhook. The -a flag indicates that the webhook should be of the validating variety. Kubernetes supports other types of webhooks, which is why we need to specify -a here.

npx kubenode add webhook -g math.io -v v1 -k SimpleMath -a

This command updates the lib/index.js file again, and also generates new code and configuration. The source code for the newly created webhook can be found in lib/webhook/math.io_v1/simplemath_validating_webhook.js. In its initial state, the webhook allows every request it receives. We are going to update the code so that it rejects invalid operation strings. Update the code as shown below:

import { webhook } from '@kubenode/controller-runtime';

export class SimpleMathValidatingWebhook {
  constructor() {
    this.path = '/validate-math-io-v1-simplemath';
  }

  handler(context, request) {
    if (request.operation === 'CREATE' || request.operation === 'UPDATE') {
      const operation = request.object.spec.operation.toLowerCase();

      if (operation !== 'add' && operation !== 'subtract' &&
          operation !== 'multiply' && operation !== 'divide') {
        return webhook.admission.denied(`Unsupported operation: ${operation}`);
      }
    }

    return webhook.admission.allowed();
  }

  setupWebhookWithManager(manager) {
    const handler = this.handler.bind(this);

    manager.getWebhookServer().register(this.path, handler);
  }
}

Build and Deploy

To run our controller on a Kubernetes cluster, we must build a container image, push it to a container registry, and deploy it to the cluster. You will need to decide where you are going to host your image. This article will use localhost:5000/controller for the image reference because that is what the minikube registry defaults to. Use the following command to update the image reference in the Kubenode project:

$ npx kubenode configure manager-image localhost:5000/controller

Next, build the image using the following command. You must have Docker installed on your system for this to work.

$ npm run docker-build

Once the build completes, push the image to your configured registry using the following command:

$ npm run docker-push

Once the push finishes, deploy all of the generated Kubernetes resources using the following command. You must have kubectl installed and configured to access a cluster in order for this to work.

$ npm run deploy

Inspecting the System

If everything worked as expected, our container should be running. You can validate this using the following command:

$ kubectl get all -n math-project

The output should look similar to the following snippet.

NAME                                      READY   STATUS    RESTARTS   AGE
pod/controller-manager-75b9c4d8bc-r9rhr   1/1     Running   0          24s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/webhook-service   ClusterIP   10.102.10.128   <none>        443/TCP   24s

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller-manager   1/1     1            1           24s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-manager-75b9c4d8bc   1         1         1       24s

You can also tail the logs from the container using the following command:

$ kubectl logs -n math-project deployment/controller-manager -f

At this point, our controller is not doing anything because we haven't created any SimpleMath resources for it to monitor. Luckily, Kubenode has generated a sample YAML file that we can use to create a SimpleMath resource. Open config/samples/math.io_v1_simplemath.yaml, and update it to match the following snippet:

apiVersion: math.io/v1
kind: SimpleMath
metadata:
  labels:
    app.kubernetes.io/managed-by: kubenode
    app.kubernetes.io/name: math-project
  name: sample-simplemath
  namespace: default
spec:
  number1: 50
  number2: 25
  operation: add

Next, use kubectl to create the sample resource and read it back. If you are still tailing the controller's logs, you should see some output there as well.

$ kubectl apply -f config/samples/math.io_v1_simplemath.yaml
$ kubectl get simplemath -n default sample-simplemath -o yaml

This is a good time to experiment with the system. As an exercise you can try creating another resource with an invalid operation in order to trigger the rejection logic of the webhook. You can also try adding more functionality such as rejecting division by zero in the webhook, or supporting more operations in the controller.

Cleanup

Finally, the system can be torn down by running the following command:

$ npm run undeploy

Conclusion

This post has shown how Kubernetes controllers can be created using Node.js and Kubenode. We stepped through the creation of a simple controller that receives inputs, performs reconciliation, and generates outputs. We also added input validation via a validating webhook and tested the system out.

While Kubenode is still a young project, it has the potential to make Kubernetes development more approachable to developers who may not be familiar with Golang, or who already use JavaScript for their entire stack. If you'd like to help out with the development of Kubenode, contributions of all types are very welcome!