Kubernetes-Native CI/CD With Tekton and ArgoCD

Introduction

Having learned about the DigitalOcean Kubernetes Challenge, I decided to take it on to upskill myself on trending DevOps tools for Kubernetes, and to evaluate DigitalOcean's container services. For the challenge, I implemented a Kubernetes-native CI/CD solution using Tekton build and ArgoCD for deployment. In this article, I will share my experience and sample code I've put together, so you too can try it out.

About the sample code

You can find the sample code for the challenge at github.com/acwwat/do-k8s-gitops. The folder structure is as follows:

The app folder contains the source code for the sample to-do list application that we will build into Docker images and deployed to Kubernetes. The frontend todo-list-frontend is a basic Vue.js application and the backend todo-list-backend is a basic Java Spring Boot application with an embedded H2 database.
The config folder contains the starter Kubernetes YAML file template which we will use to deploy the application to the Kubernetes cluster.
The tekton folder contains the Tekton Kubernetes YAML files that defines the various components that form the CI pipeline.

Clone or download this GitHub repository as we will need it to set up the CI/CD solution.

Creating the application and config repositories

You need to create two repositories - one for the application to drive CI and the other for the Kubernetes configuration to drive CD. I used GitHub myself, but you can also use other Git repositories so long as they are accessible by Tekton and ArgoCD.

For the application repository, check in the content of the app folder from the sample code repository.

For the configuration repository, check in the content of the config folder. Since the Kubernetes YAML file requires a hardcoded container image path from your DigitalOcean Container Registry, you need to copy todo-list/todo-list.yaml.template as todo-list/todo-list.yaml and replace the URL of your container registry once you have created it.

Creating access tokens for integration

You need to create the following access tokens for integration purposes:

A GitHub personal access token for the Tekton pipeline to update the Kubernetes YAML files in the configuration repository. The PAT must have all scopes under the repo category selected.
A DigitalOcean personal access token for the Tekton pipeline to push the application Docker images to the DigitalOcean Container Registry. The PAT must have the write scope selected.

Copy the tokens and set them aside for now, as you will later store them as Kubernetes secrets.

Installing CLI tools

To keep things simple, I chose to favor GUIs over CLI tools for this challenge. However you still need to install a few CLI tools as follows:

Creating the Kubernetes cluster in DigitalOcean

Setting up a Kubernetes cluster can be very involved, but DigitalOcean has made it simple. Once you have your account and logged into the dashboard, create a project and then create a Kubernetes cluster. For the challenge, I used the following settings:

Create cluster

After the Kubernetes cluster is provisioned after several minutes, install the NGINX Ingress Controller as a 1-Click App on the cluster details page:

Deploy NGINX Ingress Controller

An ingress controller is not a must, however I'd like to evaluate it on DigitalOcean for practical purposes. Allow it a few minutes to deploy. When it is done, run the doctl command from the cluster details page to connect kubectl to your cluster:

Connect kubectl to the cluster

Creating the Kubernetes Container Registry

You need a container registry to store the application container images, so this is our chance to also evaluate the DigitalOcean Container Registry. In the DigitalOcean Dashboard, create a registry with the Basic subscription plan, as we require two repositories (one per Docker image) in the registry. When the registry is created, enable integration to your Kubernetes cluster in the settings.

Don't forget to create the Kubernetes YAML file in the configuration Git repository as mentioned earlier!

Installing Tekton

Tekton is an open-source, Kubernetes-native CI/CD framework. Tekton itself is a Kubernetes application and it provides custom resources as building blocks for pipelines. It took me a while to get past the finer granularity vs. GitHub Actions and Azure Pipelines, but I also appreciate its flexibility and tight integration with Kubernetes. In this challenge, I chose to implement CD using ArgoCD, however using Tekton for CD is also perfectly doable.

Installing Tekton is relatively simple. Follow the Getting Started guide to install the core components and set up the ConfigMaps for persistent volumes. You can also install the Tekton CLI and run the sample task if you like, but we won't use the CLI in this exercise. By default, Tekton uses the default service account in the cluster to run pipelines. While it would be a security best practice to create a separate service account for CI/CD, we will just keep the default for now.

As well, install the Tekton Dashboard which we will use to view pipeline progress. To not expose the Tekton Dashboard to the internet, we can use the port forwarding option to map a local port to the dashboard HTTP port. Open a new command prompt and run the following command:

kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097

Then open http://localhost:9097 in a web browser and verify that the dashboard is accessible.

Verify Tekton installation

Installing ArgoCD

Now let's switch gear to ArgoCD. ArgoCD is a declarative, GitOps CD tool for Kubernetes. Its concept is simple - detect changes between current state in the Kubernetes cluster and desired state per configuration in Git, and apply them either on-demand or automatically. With configurations in Git, various DevOps best practices such as version control, code review, and CD are possible.

Installing ArgoCD is also very simple - just follow the Getting Started guide. We won't need the CLI for the exercise, but feel free to install it when going through the guide. Once installation completes, you can forward the server port to access the ArgoCD UI without exposing it to the internet. Open a new command prompt and run the following command:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Then open http://localhost:8080 in a web browser and log in as admin using the initial password from the secret (or the updated password if changed via the CLI), then verify that the dashboard is accessible.

Verify ArgoCD installation

Deploying the Tekton CI pipeline

Now we are ready to deploy the Tekton CI pipeline for the sample application. Before you start, take a look at the Concepts page in the Tekton documentation to understand what the building blocks are and how they interact with one another. But in essence, we need to know the following:

A pipeline consists of tasks to perform the required build operations.
A task consists of steps to encapsulate a build operation, such as cloning a Git repository or updating Kubernetes YAML files in the configuration Git repository.
Tekton provides a catalog of shared tasks and pipelines that can be used out of the box.
A step is an operation run in a container in the Kubernetes cluster.
A volume-based workspace is used by tasks and steps to share data across the pipeline

Coming from a traditional CI/CD world of Jenkins and such, running a step in a container seems awfully inefficient to me. However it is not the case, since containers spin up relatively quickly and Tekton handles the lifecycle anyway.

The CI pipeline for our sample to-do list application consists of the following tasks:

Use the git-clone task from the Tekton catalog to clone the application repository into a (shared) workspace.
Use the kaniko task from the Tekton catalog to build both todo-list-frontend and todo-list-backend using the provided Dockerfiles, and push the resulting images to the DigitalOcean Container Registry. Kaniko, by the way, is a tool to build container images inside a Kubernetes cluster.
Use a custom task we put together to clone the configuration repository, update the container image version in the Kubernetes YAML file, and commit the changes to Git (for ArgoCD to pick up).

Let's now set up the Tekton pipeline for the sample application.

Setting up the prerequisites

Perform the following steps to provision everything the pipeline needs in the Kubernetes cluster. There are many steps, so hang in there!

Create a namespace called todo-list-ci to host the Tekton pipeline resources with the following command:
```
 kubectl create namespace todo-list-ci
```
Create a secret that the custom task uses to commit the updated Kubernetes YAML file to the configuration repository on GitHub. Since the step uses git command with no user interaction, I decided to use the GIT_ASKPASS environment variable to provide the PAT after reading this article as a quick and dirty solution. Create a new YAML file using the following template (see also tekton/config-git-askpass.yaml.template in the sample code repository) and your GitHub PAT previously created:
```
 apiVersion: v1
 kind: Secret
 metadata:
     name: config-git-askpass
 type: Opaque
 stringData:
     .git-askpass: |
         echo "### Your GitHub PAT goes here ###"
```
Then create the secret with the following command:
```
 kubectl apply -f config-git-askpass.yaml -n todo-list-ci
```
Create a secret that provides the kaniko task a Docker config to push the application container images to our DigitalOcean Container Registry. Note that the kaniko task expects the file name to be config.json, which is different from the default file names that Kubernetes expects. In any case, create a new YAML file using the following template (see also tekton/do-docker-config.yaml.template in the sample code repository) and your DigitalOcean PAT previously created:
```
 apiVersion: v1
 kind: Secret
 metadata:
     name: do-docker-config
 type: kubernetes.io/dockercfgjson
 data:
     config.json: |
         ### Your DigitalOcean PAT goes here ###
```
Then create the secret with the following command:
```
 kubectl apply -f do-docker-config.yaml -n todo-list-ci
```

Install the git-clone and kaniko tasks from the Tekton catalog with the following commands:

 kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.5/git-clone.yaml -n todo-list-ci
 kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kaniko/0.5/kaniko.yaml -n todo-list-ci

The pipeline requires a shared workspace to store the source code and configuration from Git. In the PipelineRun configuration, we need to provide a persistent volume claim (PVC) to provision the workspace. The YAML file is as follows (see also tekton/todo-list-ci-pvc.yaml in the sample code repository):
```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: todo-list-ci-pvc
 spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 5Gi
```
To create the PVC, run the following command:
```
 kubectl apply -f  todo-list-ci-pvc.yaml -n todo-list-ci
```

Installing the custom task resource

Since there is no readily available task to update the Kubernetes YAML file with new image versions, I had to create a custom task. Drawing inspiration from Sebastian Daschner's example, the resulting task resource is defined as follows (see also tekton/update-config-task.yaml in the sample code repository):

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: update-config
spec:
  params:
  - name: buildVersion
  - name: gitUrl
  - name: gitBranch
  - name: k8sYaml
  - name: gitUserName
  - name: gitUserEmail
  workspaces:
  - name: git-source
  - name: git-askpass-secret
  steps:
  - name: git-checkout
    image: alpine/git:v2.26.2
    workingDir: "$(workspaces.git-source.path)"
    script: |
      #!/usr/bin/env sh
      set -e
      cp $(workspaces.git-askpass-secret.path)/.git-askpass ~/
      chmod +x ~/.git-askpass
      export GIT_ASKPASS=~/.git-askpass
      rm -rf config
      git clone -b $(inputs.params.gitBranch) $(inputs.params.gitUrl) config
  - name: update-yaml
    image: alpine/git:v2.26.2
    workingDir: "$(workspaces.git-source.path)"
    script: |
      #!/usr/bin/env sh
      set -e
      cd config
      sed -i "s#/todo-list-frontend:[a-zA-Z0-9.]\\+#/todo-list-frontend:$(inputs.params.buildVersion)#" $(inputs.params.k8sYaml)
      sed -i "s#/todo-list-backend:[a-zA-Z0-9.]\\+#/todo-list-backend:$(inputs.params.buildVersion)#" $(inputs.params.k8sYaml)
      cat $(inputs.params.k8sYaml)
  - name: commit-push-changes-gitops
    image: alpine/git:v2.26.2
    workingDir: "$(workspaces.git-source.path)"
    script: |
      #!/usr/bin/env sh
      set -e
      cd config
      cp $(workspaces.git-askpass-secret.path)/.git-askpass ~/
      chmod +x ~/.git-askpass
      export GIT_ASKPASS=~/.git-askpass
      git config --global user.email "$(inputs.params.gitUserEmail)"
      git config --global user.name "$(inputs.params.gitUserName)"
      git add .
      git commit --allow-empty -m "[tekton] Set deployment to version $(inputs.params.buildVersion)"
      git push origin $(inputs.params.gitBranch)

As you can see, there is a common workspace used by all three steps, each of which runs a different container and script. The task is also using the secret that contains the GitHub PAT as a file that the GIT_ASKPASS environment variable refers to. As for the logic, the task first clones the configuration repository on GitHub, then uses sed to update the container image version in todo-list.yaml, and finally commits/pushes the change into Git.

Now that you understand how this custom task works, create it in your Kubernetes cluster with the following command:

kubectl apply -f  update-config-task.yaml -n todo-list-ci

Installing the pipeline resource

Lastly, we need to install the pipeline resource. Refer to the definition below (see also tekton/todo-list-ci-pipeline.yaml in the sample code repository):

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: todo-list-ci-pipeline
spec:
  workspaces:
    - name: git-source
    - name: do-docker-config-secret
    - name: config-git-askpass-secret
  params:
    - name: buildVersion
    - name: appGitUrl
    - name: appGitBranch
      default: main
    - name: frontendPathToContext
    - name: frontendImageUrl
    - name: backendPathToContext
    - name: backendImageUrl
    - name: configGitUrl
    - name: configGitBranch
      default: main
    - name: k8sYaml
    - name: gitUserName
    - name: gitUserEmail
  tasks:
    - name: clone-repo
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: git-source
      params:
        - name: url
          value: "$(params.appGitUrl)"
        - name: revision
          value: "$(params.appGitBranch)"
        - name: subdirectory
          value: "app"
        - name: deleteExisting
          value: "true"
    - name: frontend-build-image
      taskRef:
        name: kaniko
      runAfter:
        - clone-repo
      workspaces:
        - name: source
          workspace: git-source
        - name: dockerconfig
          workspace: docker-config-secret
      params:
        - name: CONTEXT
          value: app/$(params.frontendPathToContext)
        - name: IMAGE
          value: $(params.frontendImageUrl):$(params.buildVersion)
    - name: backend-build-image
      taskRef:
        name: kaniko
      runAfter:
        - clone-repo
      workspaces:
        - name: source
          workspace: git-source
        - name: dockerconfig
          workspace: do-docker-config-secret
      params:
        - name: CONTEXT
          value: app/$(params.backendPathToContext)
        - name: IMAGE
          value: $(params.backendImageUrl):$(params.buildVersion)
    - name: update-config
      taskRef:
        name: update-config
      runAfter:
        - frontend-build-image
        - backend-build-image
      workspaces:
        - name: git-source
          workspace: git-source
        - name: git-askpass-secret
          workspace: config-git-askpass-secret
      params:
        - name: buildVersion
          value: "$(params.buildVersion)"
        - name: gitUrl
          value: "$(params.configGitUrl)"
        - name: gitBranch
          value: "$(params.configGitBranch)"
        - name: k8sYaml
          value: "$(params.k8sYaml)"
        - name: gitUserName
          value: "$(params.gitUserName)"
        - name: gitUserEmail
          value: "$(params.gitUserEmail)"

The definition is a tad long, but it performs the tasks as explained earlier. Let's create it with the following command:

kubectl apply -f  todo-list-ci-pipeline.yaml -n todo-list-ci

Now that our Tekton CI pipeline is ready to go, feel free to take a well-deserved break and digest all the information we've gone through so far.

Create an application in ArgoCD

At last, we need to create the CD pipeline to complete our solution. Luckily this is very simple to do in the ArgoCD UI. You could use a declarative setup to see full benefits, but let's take it easy for now. Follow the steps below to create the application:

Log in to the ArgoCD UI.
Click the New App button.
In the General section, enter an application name and select the default project.
Scroll down to the Source section and enter your configuration repository URL and todo-list for the path.
Scroll down to the Destination section and select https://kubernetes.default.svc as the cluster URL (this is the value for deploying to the same cluster in which ArgoCD is running) and enter todo-list as the namespace. Click the Create button to create the app.

The todo-list app is now created with the missing and OutOfSync status:

New app status

Don't sync it just yet because we haven't deployed the container images to the container registry! We will get the container images ready by running the CI pipeline.

Running the Tekton CI pipeline

There are two ways to run a pipeline in Tekton:

Creating a PipelineRun Kubernetes custom resource
Configuring a Tekton Trigger to automatically start pipeline runs

Triggers can be overwhelming for beginners, so we will just manually create a PipelineRun resource to start the pipeline for this exercise. Create a new YAML file using the following template (see also tekton/todo-list-ci-pipelinerun.yaml.template in the sample code repository) and replace the values with ones specific to your environment:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: todo-list-ci-
spec:
  pipelineRef:
    name: todo-list-ci-pipeline
  params:
    - name: buildVersion
      value: "0.0.1"
    - name: appGitUrl
      value: ### Your app Git repo URL goes here ###
    - name: appGitBranch
      value: ### Your app Git branch name goes here (typically main or master) ###
    - name: frontendPathToContext
      value: "todo-list-frontend"
    - name: frontendImageUrl
      value: "### Your DigitalOcean Container Registry URL goes here ###/todo-list-frontend"
    - name: backendPathToContext
      value: "todo-list-backend"
    - name: backendImageUrl
      value: ""### Your DigitalOcean Container Registry URL goes here ###/todo-list-backend"
    - name: configGitUrl
      value: ### Your config Git repo URL goes here ###
    - name: configGitBranch
      value: ### Your config Git branch name goes here (typically main or master) ###
    - name: k8sYaml
      value: "todo-list/todo-list.yaml"
  serviceAccountName: default
  workspaces:
    - name: git-source
      persistentVolumeClaim:
        claimName: todo-list-ci-pvc
    - name: do-docker-config-secret
      secret:
        secretName: do-docker-config
    - name: config-git-askpass-secret
      secret:
        secretName: config-git-askpass

To start a new pipeline run, run the following command:

kubectl create -f  todo-list-ci-pipelinerun.yaml -n todo-list-ci

The command should complete with the PipelineRun ID similar to the following:

pipelinerun.tekton.dev/todo-list-ci-r2f94 created

Now, head over to the Tekton Dashboard in a web browser. Select PipelineRuns from the left menu and you will see that our pipeline is running!

Pipeline run status

Click the PipelineRun name and you will see the progress. In the screenshot below, I opened the build-and-push step under the frontend-build-image task, so we can see the progress of Kaniko building the image for the frontend application.

Pipeline run details

When this is done, verify in your configuration GitHub repository that a commit has been made to update the container image versions to 0.0.1. Next we can verify the deployment in ArgoCD!

Synchronizing the application in ArgoCD for the first time

Head over to the ArgoCD UI in the web browser. Then follow these steps to manually synchronize the application manifests, which will trigger a full deployment. Click the Sync button in the todo-list application tile to open the synchronize dialog, then click the Synchronize button to start the process.

Synchronize application manifests

Wait for the status to show the green Healthy and Synced statuses, then click the tile to see the deployment details. In the application details page, you will see a visualization of the Kubernetes resources for the application and other details. To open the application, click the 3rd button inside the todo-list ingress box as highlighted in the screenshot below:

Application resource details and visualization

Finally we get to see the fruit of our labor! Feel free to play around with the to-do list application to make sure that it is working.

Verify to-do list application

Enabling auto-sync in ArgoCD to complete the CI/CD solution

As we have finally verified a deployment, we need to make one final change to complete our CI/CD solution. To truly enable GitOps and CD, we need to turn on auto-sync for the application in ArgoCD.

Back on the app details page in ArgoCD UI, click the App Details button to open the details dialog. In the Summary tab, scroll down to the Sync Policy section and click the Enable Auto-Sync button.

Enable auto-sync

Then click the OK button when the prompt appears:

Confirm auto-sync enablement

Lastly, we will trigger another run for the Tekton CI pipeline for the final verification. We could make some functional changes to the todo-list application, but to test auto-sync we can just update the image version to simulate an "application update". Edit the PipelineRun YAML file created earlier and change the buildVersion parameter value to, say, 0.0.2. Then run the following command to start a new pipeline run:

kubectl create -f  todo-list-ci-pipelinerun.yaml -n todo-list-ci

Check the pipeline progress in Tekton Dashboard. When the run completes, go back to ArgoCD UI and verify that it has automatically synchronize after on the Git changes. You may need to refresh the status in the UI to reflect the latest status. In the application details page, the current sync status now shows that version 0.0.2 is deployed:

Verify app version

Congratulations! You have successfully created a basic Kubernetes-native CI/CD solution using Tekton and ArgoCD on DigitalOcean!

After you have admired your creation and played with the environment more, don't forget to destroy the Kubernetes cluster in DigitalOcean and any leftover resources (volumes and load balancers for the ingress) to avoid unnecessary cost!

What's next?

In this challenge, I have only scratched the surface of Tekton and ArgoCD. With my remaining DigitalOcean credits available for another month, I would like to build upon the current solution with the following:

Implement trigger to truly complete the CI pipeline
Extend the current solution to multiple environments (dev, QA, prod) and evaluate various high availability features
Research on best practices and improve the current solution

Check back later for new posts as I explore further!

Resources

During the challenge, I found the following resources that helped me understand how to use the tools and solve unexpected issues. Credit goes to the authors and I hope this curated list helps you as well.

Kubernetes-Native Build & Release Pipelines with Tekton and ArgoCD by Adri Villela - A comprehensive series on end-to-end Kubernetes-native CI/CD, as recommended on the challenge page itself.
Tekton with ArgoCD by Sebastian Daschner - A great blueprint for CI/CD with Tekton and ArgoCD. It saved me time scripting the Git update part for this challenge.
ArgoCD Tutorial for Beginners | GitOps CD for Kubernetes by TechWorld with Nana - A well-explained video tutorial for ArgoCD on YouTube. She has a knack for explaining complex concepts and I am a big fan of her videos.
How to fix in Kubernetes – Deleting a PVC stuck in status "Terminating" - This article helped me fix a problem where I could not delete a PVC and the associated volume.
The Vanilla DevOps Git Credentials & Private Packages Cheatsheet by AJ ONeal - This cheatsheet helped me choose the best Git credentials management strategy for this challenge.

Avangards Blog