Cuestomize Introduction

Cuestomize is a Kubernetes Package Manager using CUE-lang and integrated in Kustomize.

It is implemented as a Kustomize KRM function that reads a CUE model, and optionally some input resources from the Kustomize stream, generates some manifests, and passes them back to the Kustomize stream.

It provides the type-safety of CUE and the flexibility of kustomize, combined in a single tool.

Moreover, it allows your CUE model to consume resources from the Kustomize stream, which can be used to feed the CUE model with additonal contextual data.
This means with Cuestomize you have two ways to pass input data to your CUE model:

• the input section of the KRM function’s specification (similar to Helm values)
• resources from the Kustomize stream.

The CUE model can then use the input values and resources to generate the output manifests.

The CUE model can either be pulled from an OCI registry, or be local to the KRM function (in which case you need to build and image that bundles both the CUE model and the Cuestomize binary).

Features

• Type-safety: CUE is a strongly typed language, so you are guaranteed that the generated resources are valid from a schema point of view.
• Flexibility: Kustomize is a very flexible tool, and Cuestomize lets you leverage that flexibility by being integrated in Kustomize, while still having the benefits of CUE.
• Modularity: CUE models can be composed together, so you can build complex models by combining simpler ones.
• OCI support: CUE models can be pulled from OCI registries, making it easy to share and reuse them.
• Validation: Cuestomize, leveraging CUE, gives you the power to validate your manifest generation process from end to end. You can validate both the input data and the generated resources against their respective schemas.
  With Cuestomize, you won’t suffer from YAML indentation issues or misspelled fields anymore. If an unexpected field is found, or a required field is missing, CUE will raise an error during the evaluation of the model, preventing the generation of invalid manifests.

How it Works

Cuestomize is implemented as a Kustomize KRM function. It reads its configuration (and optionally other manifests) from the Kustomize input stream, unifies them with a CUE model of your choice, collects the model’s outputs, and passes them back to Kustomize.

Visual Example Representation

A practical example of how data are passed and manifests generated is shown below.

Say that you have a CUE model that generates a ConfigMap with data taken from the input values, and a Deployment and a Service that are expected to be present in the Kustomize input stream.

Input Stream

On the left side of Figure 2, you can see the Kustomize input stream, which contains:

• the Cuestomize KRM function configuration, which specifies the CUE model to use, the input values to pass to the model, and which resources from the stream to forward to the model
• the other manifests from the Kustomize input stream.

CUE Model Unification

In Figure 2, on the center-top, you can see the CUE model that the function will use to generate the manifests.

In the center, you can see how the unified CUE model – i.e. the resulting CUE configuration after inputs and includes are forwarded to the model – would look like:

• the input field contains the input values forwarded from the function configuration
• the includes field contains the resources forwarded from the Kustomize input stream, in a map for ease of access
• the outputs field contains the generated resources, the ConfigMap in this case, which will be collected and passed back to Kustomize by Cuestomize.

Output Stream

On the right side of Figure 2, you can see the manifests that are collected by the function, only the ConfigMap in this case. The outputs manifests are then passed back to Kustomize, which can further process them.

Getting Started

This section will guide you through the steps to get started with Cuestomize.

The CUE Model

To get started with Cuestomize, you need to create a CUE module that defines your manifest generation logic.

You can either create your own CUE model from scratch, or use an existing one.

How to create CUE models that are compatible with Cuestomize is explained in different sections of this book, so we will use one of the existing models for this example.

The Kustomization

You need to have a Kustomization directory that you can use to run Kustomize. In this directory, you need to create a kustomization.yaml file that defines the resources you want to manage with Kustomize.

How to create a Kustomization project is out of the scope of this book, so we will assume you already have one, and the next steps will assume you are using the one under examples/simple/kustomize.

In your kustomization directory, you need to add a file that holds the configuration of the KRM function that will run Cuestomize. The name you give to this file is not important, as long as you reference it in the transformers section of your kustomization.yaml file.

Create the KRM Function Configuration File

Change directory, and create a file named krm-func.yaml in the kustomization directory:

# We assume you have git cloned the repo locally and are in the root directory of the repo
cd examples/simple/kustomize

touch krm-func.yaml

Note: the name you give to this file is not important, as long as you reference it in the transformers section of your kustomization.yaml file.

Update the Kustomization File

Edit the kustomization.yaml file to add the krm-func.yaml file to the transformers section:

kind: Kustomization

# ... other sections ...

transformers:
  - krm-func.yaml

Configuring the KRM Function

Edit the krm-func.yaml file to configure the KRM function that will run Cuestomize.

Here is an example configuration:

apiVersion: cuestomize.dev/v1alpha1
kind: Cuestomization
metadata:
  name: example
  annotations:
    config.kubernetes.io/local-config: "true"
    config.kubernetes.io/function: |
      container:
        image: ghcr.io/workday/cuestomize:latest
        network: true
input:
  configMapName: example-configmap
includes:
  - group: apps
    version: v1
    kind: Deployment
    name: example-deployment
    namespace: example-namespace
  - version: v1
    kind: Service
    name: example-service
    namespace: example-namespace
remoteModule:
  ref: ghcr.io/workday/cuestomize/cuemodules/cuestomize-examples-simple:latest

Note: Cuestomize does not constrain the apiVersion and kind fields of the KRM function configuration, so you can use whatever values you want, as long as they are valid Kubernetes resource names. In your CUE model, on the other hand, you can constrain these to specific values in order to ensure compatibility between the model and the function’s configuration.

Running Kustomize to Generate the Manifests

Now that you have everything set up, you can run Kustomize to generate the manifests.

Since Cuestomize is a KRM function, you’ll need a few extra flags in order for kustomize build to work properly:

• --enable-alpha-plugins to enable the KRM function
• --network if your CUE model is pulled from a registry (can be omitted if the model is local to the function’s image).

Build the Manifests

kustomize build . --enable-alpha-plugins --network

You should see the build to be successful, and the CUE-generated manifests within the printed manifests.

Function Configuration Reference

This section documents all configurable fields for a Cuestomize KRM function configuration.

KRM Function Configuration

Metadata

The metadata field of the configuration must contain some annotations in order for kustomize to recognise it as a KRM function.
On top of that, Cuestomize offers some configurations options through the .metadata field.
All these options are documented below.

Annotations

.metadata.annotations

Annotation – config.kubernetes.io/function

The annotation config.kubernetes.io/function is the one used by kustomize to configure a KRM function (kustomize docs).

Its value must contains the configuration for the container that runs the KRM function.

metadata:
  name: my-config
  annotations:
    config.kubernetes.io/function: |
      container:
        # the Cuestomize image you want to use
        image: ghcr.io/workday/cuestomize:latest

        # this is required to pull the CUE module from a registry
        network: true

⚠️ Passing environment variables to KRM functions is a discouraged practice (and may be removed in future kustomize versions), but is documented here for completeness. It also may be useful when developing to quickly iterate, without having to change the configuration.

The KRM function configuration also accepts environment variables to be passed to the container running the function, although that is discouraged and may be removed in future kustomize versions.

Cuestomize allows you to configure the logging level and pass the credentials for private registries through environment variables.

Annotation – config.cuestomize.io/validator

Setting config.cuestomize.io/validator: "true" in the configuration annotations tells Cuestomize to use the CUE module as a validator only: it will unify the inputs and includes with the module, but it won’t collect the outputs.

This is useful if you want to validate a set of manifests with some CUE constraints, e.g. ensuring that all Deployments use a particular securityContext, or that resources in certain namespaces has a particular label, etc.

When used in validator mode, CUE will be used to validate, instead of to generate, and the behaviour you can expect is the same as running cue eval command.

Input

Input is an object whose shape depends on the CUE model you are integrating with.

The CUE model creator (yourself, or a third party) defines the shape of the input directly in the model itself.

The allowed shape and values for the input are entirely dependent on the CUE model constraints the model imposes.

For example, if the CUE model defines the input as:

input: {
    name: string
    replicas: int | *1
    region: "us-west-1" | "us-east-1" | "eu-central-1"
}

the input section of the KRM function configuration must conform to that shape:

# valid input section
input:
  name: my-app
  replicas: 3
  region: us-west-1

On the other hand, the following input would be invalid, and Cuestomize would raise an error at runtime:

# invalid input section
input:
  name: my-app
  replicas: "three" # invalid: defined as integer in the CUE model
  region: ap-southeast-1 # invalid: not one of the allowed values

As you can see, the input section is entirely dependent on the CUE model, if you change the model, the shape of the input must be updated accordingly.

Includes

The includes field is the one that ties with the concept of includes in Cuestomize. Includes, in Cuestomize, are an advanced feature that allows you to forward resources from the Kustomize input stream to the CUE model, to be used as additional input.

The power lies in the ability to let CUE “infer” some values from the kustomize stream, without forcing the user to pass them explicitly through the input section. For example, you may want to forward the Namespace resource from the kustomize stream to the CUE model, so that the model can use the received Namespace’s name to set the namespace of the generated resources, or to read and use some labels or annotations from it.

The includes field is a list of resource selectors, and resources matching one of the selectors will be forwarded to the CUE model in the includes field.

Remote Module

Auth

TODO: add description

TODO: fill this table when we support auth secrets

Example

apiVersion: cuestomize.io/v1
kind: CuestomizeConfig
metadata:
  name: my-config
  annotations:
    config.kubernetes.io/function: |
      container:
        image: ghcr.io/workday/cuestomize:latest
        network: true
remoteModule:
  ref: docker.io/wackoninja/cuemodules:latest
includes:
  - version: "v1"
    kind: ConfigMap
    name: "test-configmap"
    namespace: "test-namespace"

Pull From OCI Registry

Cuestomize can fetch CUE modules directly from an OCI registry (such as Docker Hub or GitHub Container Registry). This allows you to share and reuse CUE logic across projects and teams, and makes distributing your CUE models easier.

To pull a CUE module from an OCI registry, specify the remoteModule field in your Cuestomize KRM configuration.

Both public and private registries are supported.

Module Pull From Public Registries

Module Pull From Public Registries

ⓘ No auth is required for public modules.

To pull from a public registry, you just need to configure the function on where the CUE module to pull is stored, and which tag you want to pull. You can do this by specifying a remoteModule field in your Cuestomization resource, like so:

apiVersion: cuestomize.dev/v1alpha1
kind: Cuestomization
metadata:
  name: example
  annotations:
    config.kubernetes.io/local-config: "true"
    config.kubernetes.io/function: |
      container:
        image: ghcr.io/workday/cuestomize:latest
        network: true
input:
  configMapName: example-configmap
remoteModule:
  ref: ghcr.io/workday/cuestomize/cuemodules/cuestomize-examples-simple:latest

In this example, we are pulling the ghcr.io/workday/cuestomize/cuemodules/cuestomize-examples-simple module at the latest tag. Just by specifying .remoteModule.ref, Cuestomize will try to pull the module from the public registry before doing its processing.

Obviously, the OCI reference pulled must be a valid CUE module in order for Cuestomize to process it correctly.

Module Pull From Private Registries (With Auth)

Module Pull From Private Registries (With Auth)

For private registries or repositories, you need to provide credentials. The recommended way is to use a Kubernetes Secret and reference it in your configuration.

Secret Passing

This method requires you to have a Kubernetes Secret in the kustomize input stream having the credentials to access the private registry. The secret does not have to be present in the final output of kustomize, you can use the config.kubernetes.io/local-config: "true" annotation to tell kustomize to use the secret only during the build phase, and not show it anywhere in the rendered manifests. It is a standard way to pass sensitive data to KRM functions in kustomize.

You need to select a Kubernetes Secret through the .remoteModule.auth field:

remoteModule:
  ref: ghcr.io/workday/cuestomize/cuemodules/cuestomize-examples-simple:latest
  auth:
    kind: Secret
    name: oci-auth

This tells Cuestomize to use the oci-auth Secret for authenticating to the registry.
The secret must be in the kustomize input stream to the function in order for it to be found and used by it.

💡 You can use Kustomize’s secretGenerator to create a Secret from environment variables:

.env file

username=<username>
password=<password>

`kustomization.yaml

secretGenerator:
  - name: oci-auth
    envs:
      - .env
    options:
      disableNameSuffixHash: true
      annotations:
        # ensures this secret is not included in the final output
        config.kubernetes.io/local-config: "true"

This will generate a Secret named oci-auth with your credentials.

Auth Secret Configuration

The following structure types are supported for the auth secret:

Support may be expanded in the future to include other types, such as Docker config files.

Structure Type – Secret

The Secret structure type expects a standard Kubernetes Secret containing the username and password fields in either the data or stringData sections.

The full list of supported fields is the following:

Environment Variables (Discouraged)

This method of passing credentials is discouraged and may be removed in future kustomize versions, but is documented here for completeness, and because it may be useful when developing to quickly iterate.

TODO: document

Developing CUE Models

We call CUE model any CUE module that can integrate with Cuestomize.

This section covers how to develop CUE models.

CUE Model Structure

When developing CUE modules that integrate with Cuestomize, it is important to first understand how Cuestomize interacts with CUE, and which structures are required in order to ensure compatibility.

Here is an example of what a CUE model would see from a given KRM configuration:

apiVersion: cuestomize.dev/v1alpha1
kind: Cuestomization
metadata:
  name: example
input:
  name: example
  annotations:
    example-annotation: example-value
  replicas: 3
includes:
  - version: v1
    kind: Namespace
    annotationSelector: app=example

The configuration above selects all Namespaces with the label app=example from the input stream, and includes them in the includes structure of the CUE module.

The above KRM configuration would provide the following structures to the CUE module (assuming there is a matching Namespace in the input stream):

apiVersion: "cuestomize.dev/v1alpha1"
kind: "Cuestomization"
metadata: {
    name: string
}

input: {
    name: example
    annotations: {
        example-annotation: example-value
    }
    replicas: 3
}

// includes structure: <apiVersion>: <kind>: <namespace>: <name>: {object}
includes: {
    "v1": {
        "Namespace": {
            "": {
                "example-namespace": {
                    metadata: {
                        name: "example-namespace"
                        annotations: {
                            "app": "example"
                        }
                    }
                }
            }
        }
    }
}

Fast Iteration With Local Modules

Kustomize allows KRM functions to mount local directories as function modules. This feature can be leveraged during CUE module development to enable fast iteration, without needing to interact with an OCI registry or bake an image.

Setting Up Local Module Mounting

To set up local module mounting, you need to create a kustomization file in the parent directory of your CUE module (kustomize constrains local mounts to be relative to the kustomization file, and you cannot go up in the directory tree).

Setup a directory structure like the following:

.
├── cue
│   └── cue.mod
│   └── main.cue
└── kustomization.yaml
└── krm-config.yaml

In this structure, the cue directory contains your CUE module files, while the kustomization.yaml file is in the parent directory.

Create a kustomization.yaml file with the following content:

resources:
  - krm-config.yaml
transformers:
  - krm-config.yaml

In your KRM configuration file (krm-config.yaml), specify the function with a mount to the local CUE module directory:

apiVersion: mymodule.cuestomize.dev/v1alpha1
kind: MyCUEFunction
metadata:
  name: example
  annotations:
    config.kubernetes.io/function: |
      container:
        image: ghcr.io/workday/cuestomize:latest
        mounts:
        - type: bind
          src: ./cue
          dst: /cue-resources # this is where the Cuestomize expects the CUE module to be
input: {} # your input configuration

Running Cuestomize with Local Modules

With docker running (needed by kustomize to run containerised functions), you can now quickly test your CUE-Kustomize integration by running:

kustomize build . --enable-alpha-plugins

Run the command in the directory where your kustomization.yaml file is located.

Unit Testing CUE Models

TODO: write section

Advanced Topics

This section covers some advanced Cuestomize features and concepts.

Includes

TODO: write section

Validator Mode

TODO: write section

Glossary

• Cuestomize: both the KRM function and the binary that gets executed in the container.
• CUE model: it is a CUE-lang module that bundles the manifest generation logic. It is expected to have an outputs field which is a slice of KRM resources, and optionally an input field and an includes field (see CUE Model Integration for more details).
• Unified CUE model: the result of merging the CUE model with the input values and resources forwarded from the Kustomize input stream.
• Inputs: the data that is passed to the CUE model via the input field of the function’s specification.
• Includes: resources that are passed to the CUE model via the includes field of the function’s specification. These resources are taken from the Kustomize input stream.
• Outputs: the resources generated by the CUE model and passed back to Kustomize.