Kubernetes Manifest

Customizable component for Kubernetes manifests

Suggest Edits

Introduction

With the help of Kubernetes Manifest Components, you can deploy raw Kubernetes Manifests in Bunnyshell.
Kubernetes Manifest components are defined and edited only from the bunnyshell.yaml file. Bunnyshell allows you to declare a set of bash commands to be ran when deploying, starting, stopping, or deleting the component.

📘

Note

Bunnyshell will not build the images for these components automatically, so you will need to add a Docker Image Component, or use one of your own images. See Docker Image Components and Custom Docker Image Components for more details.


Example

Let's start by looking at an example.

-
    kind: KubernetesManifest
    version: v1
    name: myapp
    runnerImage: 'alpine/k8s:1.22.15'
    deploy:
        - 'kustomize create --autodetect --recursive --labels=app.kubernetes.io/instance-myapp:bns,app.kubernetes.io/part-of:env-{{ env.unique }} --namespace {{ env.k8s.namespace }}'
        - 'kustomize edit set image nginx={{ components.myapp-image.image }}'
        - |
            kustomize edit add patch --kind Deployment --name my-app --patch '[{"op": "add", "path": "/spec/template/spec/containers/0/env/-", "value": {"name": "ENV", "value": "bunnyshell"}}]'
        - 'sed -i "s/myapp.mydomain.com/myapp-{{ env.base_domain }}/g" ingress.yaml'
        - 'kubectl apply -k .'
        - SERVICE_ENDPOINT="https://myapp-{{ env.base_domain }}/api"
    destroy:
        - 'kustomize create --autodetect --recursive --namespace {{ env.k8s.namespace }}'
        - 'kubectl delete -k .'
    start:
        - 'kubectl scale --replicas=2 --namespace {{ env.k8s.namespace }} deployment/myapp'
    stop:
        - 'kubectl scale --replicas=0 --namespace {{ env.k8s.namespace }} deployment/myapp'
    exportVariables:
        - SERVICE_ENDPOINT
    gitRepo: 'https://gitlab.com/example/my-app.git'
    gitBranch: master
    gitApplicationPath: /

📘

Please make sure to replace myapp in all occurrences if you are using this example as a base for your Components.
A common pitfall is to forget the -labels=app.kubernetes.io/instance-myapp:bns part of the kustomize create command.


Component attributes

gitRepo

The URL of the Git repository where the Kubernetes Manifest is located. The repository will be cloned in the workdir of the runner image.

gitBranch

The branch where the Kubernetes Manifest is located. This attribute is relative to the gitRepo.

gitApplicationPath

The path to the application folder, relative to the branch. When provided, it will watch for changes only within this folder to trigger Auto-updates (if enabled) or to display that a new version of your application is available.

:warning: The repository will be cloned entirely, not just the specified gitApplicationPath.

runnerImage

This is where you specify the image where Bunnyshell runs the commands given by the user. It can be the actual image of the component, or any other image.

If the value provided for runnerImage is '@self', Bunnyshell will reference the image of the component.

Commands

In the Deploy, Start, Stop and Delete properties you can add any type of commands. As mentioned before, you can deploy anywhere you want.

🚧

Note

Steps for the Deploy property are mandatory. The other properties can be left blank.

  • The deploy and start commands always run sequentially, in the order in which components are defined in bunnyshell.yaml. This is important, because we can extract one of the component’s environment variables to use it later on in a different component.
  • The stop and destroy commands run in parallel between components.

Below is an example that includes all the four properties mentioned earlier.

deploy:
        - 'kustomize create --autodetect --recursive --labels=app.kubernetes.io/instance-my-app:bns,app.kubernetes.io/part-of:env-{{ env.unique }} --namespace {{ env.k8s.namespace }}'
        - 'kustomize edit set image nginx={{ components.my-app-image.image }}'
        - |
            kustomize edit add patch --kind Deployment --name my-app --patch '[{"op": "add", "path": "/spec/template/spec/containers/0/env/-", "value": {"name": "ENV", "value": "bunnyshell"}}]'
        - 'sed -i "s/my-app.mydomain.com/my-app-{{ env.base_domain }}/g" ingress.yaml'
        - 'kubectl apply -k .'
        - SERVICE_ENDPOINT="https://my-app-{{ env.base_domain }}/api"
    destroy:
        - 'kustomize create --autodetect --recursive --namespace {{ env.k8s.namespace }}'
        - 'kubectl delete -k .'
    start:
        - 'kubectl scale --replicas=2 --namespace {{ env.k8s.namespace }} deployment/my-app'
    stop:
        - 'kubectl scale --replicas=0 --namespace {{ env.k8s.namespace }} deployment/my-app'

Displaying Components' resources in the UI

For Bunnyshell to be able to discover your Kubernetes resources, they need to have the labels below set on them. Make sure you replace the COMPONENT NAME section with the actual name of your component.

- app.kubernetes.io/part-of = env-{{ env.unique }}
- app.kubernetes.io/instance-<COMPONENT NAME> = bns

Exposing hosts

If you are using Ingress resources to expose your services from your components, you need to update the (fixed) hosts on these Ingress resources to the (dynamic) values of this pattern. Replace SERVICE with a name of your choosing, and feel free to use any replacement method you prefer, sed is just an example to get you started.

'sed -i "s/web-self-dev-test.mydomain.com/[SERVICE]-{{ env.base_domain }}/g" ingress.yaml'

Value Interpolation for Docker Images

If you are using Bunnyshell to build your images using Docker Image, you can use value interpolation as exemplified below. NAME represents the component name of your Docker Image Component. Bunnyshell will replace it with the built image.

  • {{ components.NAME.image }}: contains full image name, including the tag.
    Example: nginx:latest
  • {{ components.dev_test_image.imageName }}: contains only the image name.
    Example: nginx
  • {{ components.dev_test_image.imageTag }}: contains only the image tag.
    Example: latest

Injecting Bunnyshell Variables in your resources

If you need to inject Bunnyshell environment variables in your resources, please refer to the Value Interpolation page.

- |
    kustomize edit add patch --kind Deployment --name man-web-self \
      --patch '[{"op": "add", "path": "/spec/template/spec/containers/0/env/-", "value": {"name": "ENV", "value": "bunnyshell"}}]'

exportVariables

Using this property you can export variables from Kubernetes Manifest Components to other components inside the environment. They are captured only during the deploy and can be used after they reach the next interpolation context in the same deploy workflow, or in subsequent start/stop/destroy workflows.

📘

Note

These are not environment variables, but variables exported from your deployment script.

Example: In the section presented above, under deploy, we defined the variable SERVICE_ENDPOINT.

deploy:
        - 'kustomize create --autodetect --recursive --labels=app.kubernetes.io/instance-my-app:bns,app.kubernetes.io/part-of:env-{{ env.unique }} --namespace {{ env.k8s.namespace }}'
        - 'kustomize edit set image nginx={{ components.my-app-image.image }}'
        - |
            kustomize edit add patch --kind Deployment --name my-app --patch '[{"op": "add", "path": "/spec/template/spec/containers/0/env/-", "value": {"name": "ENV", "value": "bunnyshell"}}]'
        - 'sed -i "s/my-app.mydomain.com/my-app-{{ env.base_domain }}/g" ingress.yaml'
        - 'kubectl apply -k .'
        - SERVICE_ENDPOINT="https://my-app-{{ env.base_domain }}/api"

Normally, this variabile would only run here. But in our case, we are going to list it under exportedVariables.

exportVariables:
            - REDIS_IP

This way, after deployment Bunnyshell will store the variable (encrypted) in a database. The variable will be available for you to use in another component, if need be. You can use the exported variables in interpolation contexts using key with the format below:

{{components.NAME.exported.VAR_NAME}}

Kubernetes Manifest Components also have Environment Variables. Environment Variables are injected in runnerImage.

Usually, environment variables are injected in Kubernetes, where Bunnyshell starts your pod with the image it built. All Component, Environment and Project variables are inherited.


Resources

Resources created by KubernetesManifest Components are self-managed by the user. Bunnyshell does not analyze the Kubernetes manifests, nor does it keep track of created resources.


Resource cleanup

The cleanup should be handled manually, by defining destroy scripts.

An example on how you can achieve this is illustrated below:

kind: KubernetesManifest
name: k8s-manifest
gitRepo: 'https://gitlab.com/[...].git'
gitBranch: master
gitApplicationPath: /manifests
runnerImage: 'alpine/k8s:1.22.15'
deploy:
    - 'cd manifests'

    # create the deploy version variable
    - |
        DEPLOY_VERSION='{{ "now"|date("Ymd-His") }}'; echo $DEPLOY_VERSION
        
    # create the namespace variable
    - |
        NAMESPACE='dev-test-{{ env.unique }}'; echo $NAMESPACE

    # just a dummy Pod, always with a random name, to demonstrate it will be cleaned up on the next deploy
    - |
        cat << EOF > dummy_pod.yaml
          apiVersion: v1
          kind: Pod
          metadata:
            name: dummy-pod-{{ random(10-100) }}
          spec:
            containers:
              - name: busybox
                image: busybox:1.34.1
                command: [ "/bin/sh", "-c", "tail -f /dev/null" ]
        EOF

    - 'kubectl create ns $NAMESPACE || true'
    
    # start the kustomization.yaml file by adding app.kubernetes.io/instance-k8s-manifest and app.kubernetes.io/part-of commonLabels to all resources in this folder, to see the k8s resources in Bunnyshell UI
    
    - 'kustomize create --autodetect --recursive --labels=app.kubernetes.io/instance-k8s-manifest:bns,app.kubernetes.io/part-of:env-{{ env.unique }} --namespace=$NAMESPACE'
    # tweak the kustomization.yaml to add the app.bunnyshell.com/deploy-version label to all the targeted resources (you can choose whatever label name you want for this)
    - |
        cat << EOF >> kustomization.yaml
        labels:
          - pairs:
              app.bunnyshell.com/deploy-version: $DEPLOY_VERSION
            includeSelectors: false
        EOF
        
    # add the app.bunnyshell.com/deploy-version also to Deployment Pods, you need this for cleanup
    - |
        kustomize edit add patch --kind Deployment --patch '[{"op": "add", "path": "/spec/template/metadata/labels/app.bunnyshell.com~1deploy-version", "value": "'$DEPLOY_VERSION'"}]'
    
    # add the app.bunnyshell.com/deploy-version also to StatefulSet Pods if you know you have, you need this for cleanup
    - |
        kustomize edit add patch --kind StatefulSet --patch '[{"op": "add", "path": "/spec/template/metadata/labels/app.bunnyshell.com~1deploy-version", "value": "'$DEPLOY_VERSION'"}]'
    
    # add the app.bunnyshell.com/deploy-version also to DaemonSet Pods if you know you have, you need this for cleanup
    - |
        kustomize edit add patch --kind DaemonSet --patch '[{"op": "add", "path": "/spec/template/metadata/labels/app.bunnyshell.com~1deploy-version", "value": "'$DEPLOY_VERSION'"}]'

    # example of other kustomize edits, anything you need
    - 'kustomize edit set image nginx={{ components.dev_test_image.image }}'
    - 'sed -i "s/web-self-dev-test.mydomain.com/k8s-manifest-{{ env.base_domain }}/g" ingress.yaml'

    # just a look at how the final kustomization.yaml looks
    - 'cat kustomization.yaml'
    
    # apply the manifests with kustomization.yaml patches
    - 'kubectl apply -k .'
    
    # clean up all the resources you want (all,pvc,cm,...) that where deployed from here (they have the app.kubernetes.io/instance-k8s-manifest and app.kubernetes.io/part-of labels) but don't have the latest deployment version (the value of label app.bunnyshell.com/deploy-version is not the same with the current generated one)
    - 'echo "cleaning up old resources..." && kubectl delete all,pvc,cm -l "app.kubernetes.io/instance-k8s-manifest=bns,app.kubernetes.io/part-of=env-{{ env.unique }},app.bunnyshell.com/deploy-version notin ($DEPLOY_VERSION)" --wait=false --namespace=$NAMESPACE'
destroy:
    - 'cd manifests'
    - 'kustomize create --autodetect --recursive --labels=app.kubernetes.io/instance-k8s-manifest:bns,app.kubernetes.io/part-of:env-{{ env.unique }} --namespace=dev-test-{{ env.unique }}'
    - 'kubectl delete -k .'
    - 'kubectl delete ns dev-test-{{ env.unique }}'
start:
    - 'kubectl scale --replicas=1 deployment/man-web-self -n dev-test-{{ env.unique }}'
stop:
    - 'kubectl scale --replicas=0 deployment/man-web-self -n dev-test-{{ env.unique }}'

Updated about 1 month ago