Skip to main content

Crossplane in a Box: Your Toolkit for Fast Prototyping and Testing

Published Date
Author Piotr Zaniewski

This article will be helpful for anyone interested in setting up a local Crossplane dev/test environment in a reproducible and easy way. Source code for this blog is available in a companion repository.

There are several reasons why local Crossplane environment is useful:

  • fast prototyping of the infrastructure APIs development
  • testing infrastructure changes and configuration
  • testing new Crossplane versions
  • testing new Crossplane CLI versions

Before diving any further let’s see how the dev/test environment would look like:

crossplane-in-a-box-image1

There are 3 phases that we would like to automate as much as possible:

  1. Infrastructure setup
  2. Iterative development and testing (for example developing compositions or composition functions)
  3. Optionally collaborate with others

This setup saves a lot of time, and because it’s fairly reproducible, it can be shared with team members and used as a standard template. Let's learn how to create it.

Choosing local Kubernetes cluster

First things first, in order to run Crossplane we need a Kubernetes cluster.

There are a lot of choices as it comes to running a local Kubernetes instance, but we are going to focus on KIND.

KIND is very easy to maintain and fast to set up. Second best option is k3s.

Declarative Approach All the Way?

Before we dive into the setup, let's talk about two approaches: declarative versus imperative. Both approaches makes sense depending on the circumstances. Declarative approach provides more stability and predictability, while imperative approach is suitable for quick prototyping and provides more control over the setup.

Whenever it comes to something vs something else, it's really about when it makes more sense to use something and when something else.

Given this approach, we will use both approaches depending on the requirements and what we want to achieve.

Orchestrating commands

Setting up a local or test Crossplane environment means orchestrating a bunch of commands, including scripts, YAML files, Helm charts, etc.

In the imperative paradigm, this is typically done via a Makefile or bash scripts. The problem with make is that it is designed as a tool to build C source code, it can run commands but that's not its purpose. This means that when using Makefile we take on the whole unnecessary baggage of the build part.

Separate bash scripts are a bit better but after a while they became too verbose and hard to maintain.

There is a tool that combines best of both worlds; just is similar to make, but focused on commands orchestration.

Declarative approach is still our friend

Justfile contains all imperative logic needed to quickly create and destroy our local cluster. It exposes various knobs for us to interact with it.

Installing a Helm chart, operator or simple YAML file can be done declaratively using a GitOps process. This can be accomplished with Flux or ArgoCD. For a quick, local setup ArgoCD is a bit more user friendly due to its robust web client. Here we are utilizing app of apps pattern to bootstrap additional apps from a single source. This article describes the pattern very well.

We have just scratched the surface of ArgoCD or GitOps. You can read more about GitOps in here and here.

Setup

Follow along with the companion repo.

The setup is tested on macOS and Linux. It should work on Windows with WSL2 as well.

Prerequisites

  • To learn more about setting up reproducible shell environments, watch Viktor’s video about using devbox to do just that. You can use it or setup the tools the old-fashion way.

Other than kind, we need a few additional CLI tools:

just command runner

For macOS users, you can use Homebrew to install just:

brew install just

For Linux users, refer to the just repository for installation instructions.

envsubst

There are several ways of templating YAML. We can wrap it in a Helm chart, use ytt, jsonnet, yq, kustomize or many others. Those are all valid approaches, but for local environment, there is a simpler method.
We will use envsubst instead. It is a part of the GNU gettext utilities and should be already installed on your system. This tool enables us to patch environment variables on the fly.

On macOS, you can install gettext (which includes envsubst) using Homebrew:

brew install gettext
brew link --force gettext

On Linux, envsubst is usually included with the gettext package. Use your distribution's package manager to install it:

For Debian-based distributions (e.g., Ubuntu):

sudo apt-get update
sudo apt-get install gettext

For RHEL-based distributions (e.g., CentOS):

sudo yum install gettext

kubectl

Follow the official Kubernetes documentation to install kubectl for your operating system.

With all the prerequisites installed, we can now proceed to setting up our local Crossplane environment.

How To

Here are simple steps to get started:

  1. Fork the repository https://github.com/Piotr1215/crossplane-box and clone it into your local environment
  2. Export variable with your GitHub user name export GITHUB_USER=your-github-username
  3. Run just setup to create prerequisites

Running the last command will do the following:

  • replace my GitHub user name with yours in the bootstrap.yaml and apps/application_crossplane_resources.yaml
  • create a kind cluster named control-planewith ArgoCD and Crossplane installed
  • copy ArgoCD server secret to clipboard and launch default browser with ArgoCD login page
username: admin password: should be in your clipboard so just paste it in the password text box. In case this didn't work, you can run kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d to get the password.

ℹ️ Since this setup is meant to be quick and easy, we are not setting up a HTTPS certificate for the web UI and in order to proceed click “Advanced” → “Proceed to localhost (unsafe)”.. If you’re interested in a more secure setup with HTTPS for local development, you might explore using mkcert. It requires a bit more work, as everyone needs to run mkcert on their machine to create and trust their certificates

Typing just<TAB> will show all available just recipes. Here is the list:

Available recipes:
    default        # targets marked with * are main targets
    setup          # * setup kind cluster with uxp, ArgoCD and launch argocd in browser
    setup_kind cluster_name='control-plane' # setup kind cluster
    setup_uxp xp_namespace='crossplane-system' # setup universal crossplane
    setup_argo     # setup ArgoCD and patch server service to nodePort 30950
    launch_argo    # copy ArgoCD server secret to clipboard and launch browser, user admin, pw paste from clipboard
    bootstrap_apps # bootstrap ArgoCD apps and set reconcilation timer to 30 seconds
    teardown       # * delete KIND cluster

Now we ready to start iterating quickly on our local Crossplane setup, by adding new content to the yaml directory and pushing changes to the repository. It's also possible to create a new ArgoCD app in the app folder and point it to a different repository or sub-folder.

Install more content

ArgoCD's bootstrap app observes the apps directory for any changes once deployed.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps
    repoURL: https://github.com/
/crossplane-box
    targetRevision: HEAD
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

Notice that the apps directory already contains an app that points to the yaml folder with Crossplane functions and providers. Adding to this directory and pushing the changes to the repository will automatically deploy the new content to the cluster.

When you add new content to the yaml directory, you can deploy it to the cluster by running the following command:

just bootstrap_apps

This will deploy all the apps from the apps folder and the content of the yaml folder into the cluster via ArgoCD.

  • There is no need to push content to remote repository every time files are added, simply apply the new files locally. To see the local changes in ArgoCD, use the just sync recipe after adding new ArgoCD Application to the apps folder or any other YAML files to the yaml folder. This requires having argocd CLI installed.

Navigate to the ArgoCD web interface and you should see all the resources deployed.

crossplane-in-a-box-image2

Destroy the cluster

just teardown

Collaboration

Local setup doesn't exclude collaboration. There are a few ways how we can collaborate with this setup.

  • accept PRs to our forked repository to let someone else install infra/apps on our cluster or change the setup
  • add a new ArgoCD application pointing to a repository with resources we want to reconcile on our cluster
  • use ngrok to expose local port on the internet and share our ArgoCD instance and enable someone to see state of our cluster
Read more here about using ngrok to share a local Kubernetes service over the internet.

Summary

Deploying a local/test Crossplane instance can and should be fully automated. We've seen how using declarative and imperative techniques helped us to create a fully functional cluster with ability to add Crossplane resources and configuration.

This setup can be easily transitioned to a production cluster. Additionally it's easy to keep adding new recipes that specialize in different tasks, such as testing, tracing, debugging, etc.

I'm using this setup to test UXP and develop compositions. What will you use it for?

To see this in the overall Crossplane story, check out Viktor’s book, Crossplane: The Cloud Native Control Plane. He’s a developer advocate and DevOps expert, host of the DevOps toolkit, and overall Crossplane expert. We’re offering it at no cost as a thanks to the Upbound community, so read it to learn about Crossplane inside and out!