Skip to main content

Improve Crossplane Compositions Authoring with go-templating-function

Published Date
Author Piotr Zaniewski

Introduction

Crossplane Compositions are a powerful abstraction layer that enables platform teams to create a custom API for their internal customers to simplify and standardize the infrastructure management process. The Composition authors wrestle with the complexity of cloud infrastructure and at the same time need to ensure a stable and user-friendly API surface for the platform consumers.

Historically, Compositions were intended to support only very simple resources’ manipulation to allow the API calls to patch and transform the information passed from a Claim or Composite Resource (XR) to the Composition engine. The lack of touring complete language supporting the required transformations resulted in a very verbose YAML files with lots of repetitions, increasing the toil of authoring Compositions.

With the addition of Compositions Functions in the Crossplane v.1.11 it is possible to use a programming language, like Go or Python (more to come) or in our case a capability of Go called Go-templating to enrich and transform the authoring of Compositions.

This blog will guide you through the process of rewriting a simple internal platform Composition in the Go templating style using the go-templating-function.

Use Case: Collapsing two Compositions into one

Since traditional Composition's engine does not allow using conditionals, there is no way to render one Composition or another using a parameter from a Claim. To illustrate this example, let’s look at the two Compositions that render a GCP Bucket:

Standard bucket

This very simple composition renders a single GCP bucket

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xstoragebuckets.platform-composites.upbound.io
  labels:
    provider: gcp
    type: generic
spec:
  patchSets:
... Omitted for brevity
  writeConnectionSecretsToNamespace: upbound-system
  compositeTypeRef:
    apiVersion: platform-composites.upbound.io/v1alpha1
    kind: XStorageBucket
  resources:
  - name: storagebucket
    base:
      apiVersion: storage.gcp.upbound.io/v1beta1
      kind: Bucket
      spec:
        forProvider: 
          location: us-west1
          storageClass: STANDARD
          providerConfigRef:
            name: default
    patches:
... Omitted for brevity

Versioned Bucket

Another Composition needs to be created to accommodate for a versioning setting.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xstoragebuckets.platform-composites.upbound.io
  labels:
    provider: gcp
    type: generic
spec:
  patchSets:
... Omitted for brevity
  writeConnectionSecretsToNamespace: upbound-system
  compositeTypeRef:
    apiVersion: platform-composites.upbound.io/v1alpha1
    kind: XStorageBucket
  resources:
  - name: storagebucket
    base:
      apiVersion: storage.gcp.upbound.io/v1beta1
      kind: Bucket
      spec:
        forProvider: 
          location: us-west1
          versioning:
            - enabled: true
          storageClass: STANDARD
          providerConfigRef:
            name: default
    patches:
... Omitted for brevity

In this case, we have to have two Compositions and need to use a Composition selector in a claim just for this one field.

Using go-templating-function

The two Compositions can be transformed into just one using the go-templating-function. Follow this steps

1. Install go-templating-function and function-auto-ready from the marketplace.

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-go-templating
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-go-templating:v0.2.2

---
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-auto-ready
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-auto-ready:v0.2.1

2. Convert the Composition to use the go templating. The initial conversion gives us one to one translation between the standard “Patch and Transform” style Composition and the go-templating style. We are going to improve this design.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xstoragebuckets.platform-composites.upbound.io
  labels:
    provider: gcp
    type: generic
spec:
  writeConnectionSecretsToNamespace: upbound-system
  compositeTypeRef:
    apiVersion: platform-composites.upbound.io/v1alpha1
    kind: XStorageBucket
  mode: Pipeline
  pipeline:
    - step: render-templates
      functionRef:
        name: function-go-templating
      input:
        apiVersion: gotemplate.fn.crossplane.io/v1beta1
        kind: GoTemplate
        source: Inline
        inline:
          template: |
            ---
            apiVersion: storage.gcp.upbound.io/v1beta1
            kind: Bucket
            metadata:
              annotations:
                gotemplating.fn.crossplane.io/composition-resource-name: {{ .observed.composite.resource.metadata.name }}
              name: {{ .observed.composite.resource.metadata.name }}
              labels:
                owner: {{ .observed.composite.resource.spec.parameters.owner }}
                service: {{ .observed.composite.resource.spec.parameters.service }}
            spec:
              forProvider:
                location: {{ .observed.composite.resource.spec.parameters.location }}
                storageClass: {{ .observed.composite.resource.spec.parameters.class }}
              providerConfigRef:
                  name: {{ .observed.composite.resource.spec.parameters.environment }}
    - step: ready
      functionRef:
        name: function-auto-ready

Using Composition Functions is very well documented in the Crossplane docs. Read the details to learn more about it.

The current implementation of our Composition renders the non-versioned Bucket. Next steps are to change the XRD and template to accommodate for the versioning field.

Using the new Crossplane beta trace command is helpful in checking the latest events on the composed resources from the Claim/XR:

piotr_go_templating

Adding the versioning field

1. Modify the XRD to set the versioning field with a boolean flag:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xstoragebuckets.platform-composites.upbound.io
spec:
  group: platform-composites.upbound.io
  names:
    kind: XStorageBucket
    plural: xstoragebuckets
  claimNames:
    kind: StorageBucket
    plural: storagebuckets
  defaultCompositionRef:
    name: storagebuckets.platform-composites.upbound.io
  connectionSecretKeys:
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                parameters:
                  type: object
                  description: Generic XRD parameters
                  properties:
                    versioningEnabled:
                      type: boolean
                      description: specify if the bucket should be versioned
                    owner:
                      type: string
                      description: Squad or individual who owns the cloud resource.
                    service:
                      type: string
                      description: Service resource belogs to, like shimmer, api etc.
                    location:
                      type: string
                      description: Passthrough location from cloud provider. Defaults to us-west1 for GCP.
                    environment:
                      type: string
                      description: Playground, dev, staging, production this maps to ProviderConfig that points to a specific project in GCP.
                    storageClass:
                      type: string
                      description: "Possible values: STANDARD, NEARLINE, COLDLINE. Defaults to STANDARD. The value is ingored for the secure bucket."
                  required:
                    - environment
                    - owner
                    - service
                    - versioningEnabled
              required:
                - parameters

2. Now let’s modify the Composition template to add the conditional for the versioning field.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xstoragebuckets.platform-composites.upbound.io
  labels:
    provider: gcp
    type: generic
spec:
  writeConnectionSecretsToNamespace: upbound-system
  compositeTypeRef:
    apiVersion: platform-composites.upbound.io/v1alpha1
    kind: XStorageBucket
  mode: Pipeline
  pipeline:
    - step: render-templates
      functionRef:
        name: function-go-templating
      input:
        apiVersion: gotemplate.fn.crossplane.io/v1beta1
        kind: GoTemplate
        source: Inline
        inline:
          template: |
            ---
            apiVersion: storage.gcp.upbound.io/v1beta1
            kind: Bucket
            metadata:
              annotations:
                gotemplating.fn.crossplane.io/composition-resource-name: {{ .observed.composite.resource.metadata.name }}
              name: {{ .observed.composite.resource.metadata.name }}
              labels:
                owner: {{ .observed.composite.resource.spec.parameters.owner }}
                service: {{ .observed.composite.resource.spec.parameters.service }}
            spec:
              forProvider:
                location: {{ .observed.composite.resource.spec.parameters.location }}
                storageClass: {{ .observed.composite.resource.spec.parameters.storageClass }}
                {{- if .observed.composite.resource.spec.parameters.versioningEnabled }}
                versioning:
                  - enabled: true
                {{- else }}
                 versioning:
                 - enabled: false
                {{- end }}
              providerConfigRef:
                name: {{ .observed.composite.resource.spec.parameters.environment }}
    - step: ready
      functionRef:
        name: function-auto-ready

3. Now we are ready to apply the Claim of the versioning enabled Bucket to the preconfigured GCP project.

apiVersion: platform-composites.upbound.io/v1alpha1
kind: StorageBucket
metadata:
  name: sample-storage-12345
  namespace: default
spec:
  compositionSelector:
    matchLabels:
      # Only provider GCP is available at the moment
      provider: gcp
      type: generic
  parameters:
    versioningEnabled: true
    owner: squad-platform
    service: platform-composites
    # Passthrough location from cloud provider
    # defaults to us-west1 for GCP
    location: us-west1 #Optional
    # This maps to ProviderConfig that points to a specific
    # project in GCP. Use default for local testing and playground for crossplane-playground
    environment: provider-gcp #Required
    # Possible values: STANDARD, NEARLINE, COLDLINE, ARCHIVE
    # Defaults to standard
    storageClass: STANDARD #Optional

The resource rendered correctly:

spec:
  deletionPolicy: Delete
  forProvider:
    location: us-west1
    project: squad-platform-playground
    publicAccessPrevention: inherited
    storageClass: STANDARD
    versioning:
    - enabled: true
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: provider-gcp

piotr_go_templating_2

Default values

Not all the schema fields are required, and for those the Composition provides default values. location and storageClass are not required in the schema, and the Composition provides default values for those fields when omitted. Let’s add them to the template. Now supplying a Claim without the location and storageClass fields will result in the default values being rendered like so.

...
                location: {{ default "us-west1" .observed.composite.resource.spec.parameters.location }}
                storageClass: {{ default "STANDARD" .observed.composite.resource.spec.parameters.storageClass }}
...

Bringing back PatchSets

This specific Composition is very simple, but there are multiple others where the same patch needs to be applied to multiple resources. In the original Composition we have the ownerAndServiceLabels that are applied to all resources that support labels.

- name: ownerAndServiceLabels
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.owner
          toFieldPath: spec.forProvider.labels[owner]
        - type: FromCompositeFieldPath
          fromFieldPath: spec.parameters.service
          toFieldPath: spec.forProvider.labels[service]

With the go-templating-function, it’s easy to define the patchSet-like behavior

Here is the modified Composition template part with the labels

template: |
            ---
            {{- define "ownerAndProjectLabels" }}
              labels:
                owner: {{ .observed.composite.resource.spec.parameters.owner }}
                service: {{ .observed.composite.resource.spec.parameters.service }}
            {{- end }}

            apiVersion: storage.gcp.upbound.io/v1beta1
            kind: Bucket
            metadata:
              name: {{ .observed.composite.resource.metadata.name }}
              annotations:
                gotemplating.fn.crossplane.io/composition-resource-name: {{ .observed.composite.resource.metadata.name }}
              {{ template "ownerAndProjectLabels" . }}

Template Variables

One small quality of life improvement is to define variables in the template so there is no need to type .observed.composite.resource.spec.parameters all the time. Here is the final version of the Composition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xstoragebuckets.platform-composites.upbound.io
  labels:
	provider: gcp
	type: generic
spec:
  writeConnectionSecretsToNamespace: upbound-system
  compositeTypeRef:
	apiVersion: platform-composites.upbound.io/v1alpha1
	kind: XStorageBucket
  mode: Pipeline
  pipeline:
	- step: render-templates
  	functionRef:
    	name: function-go-templating
  	input:
    	apiVersion: gotemplate.fn.crossplane.io/v1beta1
    	kind: GoTemplate
    	source: Inline
    	inline:
      	template: |
        	---
        	{{ $claim  	:= .observed.composite.resource }}
        	{{ $parameters := .observed.composite.resource.spec.parameters }}

        	{{- define "ownerAndProjectLabels" }}
          	labels:
            	owner: {{ .observed.composite.resource.spec.parameters.owner }}
            	service: {{ .observed.composite.resource.spec.parameters.service }}
        	{{- end }}

        	apiVersion: storage.gcp.upbound.io/v1beta1
        	kind: Bucket
        	metadata:
          	name: {{ $claim.metadata.name }}
          	annotations:
            	gotemplating.fn.crossplane.io/composition-resource-name: {{ $claim.metadata.name }}
          	{{ template "ownerAndProjectLabels" . }}
        	spec:
          	forProvider:
            	location: {{ $parameters.location }}
            	storageClass: {{ $parameters.storageClass }}
            	{{- if $parameters.versioningEnabled }}
            	versioning:
              	- enabled: true
            	{{ else }}
            	versioning:
              	- enabled: false
            	{{- end }}
          	providerConfigRef:
            	name: {{ $parameters.environment }}
	- step: ready
  	functionRef:
    	name: function-auto-ready

Conclusion

In conclusion, the use of the go-templating-function in Compositions authoring significantly simplifies the process of creating and managing resources with varying configurations and makes it easier to manage complex Compositions.

It allows for the creation of Compositions with conditional fields, reducing the need for multiple compositions for each variation of a resource. The re-introduction of PatchSets-like behavior makes the portability from “Patch and Transform” style Compositions easier. Adding variables scoped to Composite Resource paths makes it easier to reason about the template.

Additionally, using the new Crossplane CLI functionality to render the resources based on the Claim, Composition and functions and also trace the state of resources is a very helpful development tool. Here’s a tip: running the commands with watch makes the development loop even better.

piotr_go_templating_3

crossplane beta render examples/claim.yaml gcp-bucket/composite.yaml gcp-bucket/functions.yaml
---
apiVersion: platform-composites.upbound.io/v1alpha1
kind: StorageBucket
metadata:
  name: sample-storage-12345
---
apiVersion: storage.gcp.upbound.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/composition-resource-name: sample-storage-12345
  generateName: sample-storage-12345-
  labels:
    crossplane.io/composite: sample-storage-12345
    owner: squad-platform
    service: platform-composites
  name: sample-storage-12345
  ownerReferences:
  - apiVersion: platform-composites.upbound.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: StorageBucket
    name: sample-storage-12345
    uid: ""
spec:
  forProvider:
    location: us-west1
    storageClass: STANDARD
    versioning:
    - enabled: true
  providerConfigRef:
    name: provider-gcp

If you are interested in technical details and design of the go-templating-function, check out the go-templating-function one-pager. Big thanks to @ezgidemirel for creating the function! It significantly improves the Compositions authoring process.

You can find various functions in our Marketplace and create your own to take the Crossplane Compositions to the next level.

Read more...