Table of contents
Having learned about the DigitalOcean Kubernetes Challenge, I decided to take it on to upskill myself on trending DevOps tools for Kubernetes, and to evaluate DigitalOcean's container services. For the challenge, I implemented a Kubernetes-native CI/CD solution using Tekton build and ArgoCD for deployment. In this article, I will share my experience and sample code I've put together, so you too can try it out.
About the sample code
You can find the sample code for the challenge at github.com/acwwat/do-k8s-gitops. The folder structure is as follows:
appfolder contains the source code for the sample to-do list application that we will build into Docker images and deployed to Kubernetes. The frontend
todo-list-frontendis a basic Vue.js application and the backend
todo-list-backendis a basic Java Spring Boot application with an embedded H2 database.
configfolder contains the starter Kubernetes YAML file template which we will use to deploy the application to the Kubernetes cluster.
tektonfolder contains the Tekton Kubernetes YAML files that defines the various components that form the CI pipeline.
Clone or download this GitHub repository as we will need it to set up the CI/CD solution.
Creating the application and config repositories
You need to create two repositories - one for the application to drive CI and the other for the Kubernetes configuration to drive CD. I used GitHub myself, but you can also use other Git repositories so long as they are accessible by Tekton and ArgoCD.
For the application repository, check in the content of the
app folder from the sample code repository.
For the configuration repository, check in the content of the
config folder. Since the Kubernetes YAML file requires a hardcoded container image path from your DigitalOcean Container Registry, you need to copy
todo-list/todo-list.yaml and replace the URL of your container registry once you have created it.
Creating access tokens for integration
You need to create the following access tokens for integration purposes:
A GitHub personal access token for the Tekton pipeline to update the Kubernetes YAML files in the configuration repository. The PAT must have all scopes under the repo category selected.
A DigitalOcean personal access token for the Tekton pipeline to push the application Docker images to the DigitalOcean Container Registry. The PAT must have the write scope selected.
Copy the tokens and set them aside for now, as you will later store them as Kubernetes secrets.
Installing CLI tools
To keep things simple, I chose to favor GUIs over CLI tools for this challenge. However you still need to install a few CLI tools as follows:
Creating the Kubernetes cluster in DigitalOcean
Setting up a Kubernetes cluster can be very involved, but DigitalOcean has made it simple. Once you have your account and logged into the dashboard, create a project and then create a Kubernetes cluster. For the challenge, I used the following settings:
After the Kubernetes cluster is provisioned after several minutes, install the NGINX Ingress Controller as a 1-Click App on the cluster details page:
An ingress controller is not a must, however I'd like to evaluate it on DigitalOcean for practical purposes. Allow it a few minutes to deploy. When it is done, run the
doctl command from the cluster details page to connect
kubectl to your cluster:
Creating the Kubernetes Container Registry
You need a container registry to store the application container images, so this is our chance to also evaluate the DigitalOcean Container Registry. In the DigitalOcean Dashboard, create a registry with the Basic subscription plan, as we require two repositories (one per Docker image) in the registry. When the registry is created, enable integration to your Kubernetes cluster in the settings.
Don't forget to create the Kubernetes YAML file in the configuration Git repository as mentioned earlier!
Tekton is an open-source, Kubernetes-native CI/CD framework. Tekton itself is a Kubernetes application and it provides custom resources as building blocks for pipelines. It took me a while to get past the finer granularity vs. GitHub Actions and Azure Pipelines, but I also appreciate its flexibility and tight integration with Kubernetes. In this challenge, I chose to implement CD using ArgoCD, however using Tekton for CD is also perfectly doable.
Installing Tekton is relatively simple. Follow the Getting Started guide to install the core components and set up the ConfigMaps for persistent volumes. You can also install the Tekton CLI and run the sample task if you like, but we won't use the CLI in this exercise. By default, Tekton uses the
default service account in the cluster to run pipelines. While it would be a security best practice to create a separate service account for CI/CD, we will just keep the default for now.
As well, install the Tekton Dashboard which we will use to view pipeline progress. To not expose the Tekton Dashboard to the internet, we can use the port forwarding option to map a local port to the dashboard HTTP port. Open a new command prompt and run the following command:
kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097
http://localhost:9097 in a web browser and verify that the dashboard is accessible.
Now let's switch gear to ArgoCD. ArgoCD is a declarative, GitOps CD tool for Kubernetes. Its concept is simple - detect changes between current state in the Kubernetes cluster and desired state per configuration in Git, and apply them either on-demand or automatically. With configurations in Git, various DevOps best practices such as version control, code review, and CD are possible.
Installing ArgoCD is also very simple - just follow the Getting Started guide. We won't need the CLI for the exercise, but feel free to install it when going through the guide. Once installation completes, you can forward the server port to access the ArgoCD UI without exposing it to the internet. Open a new command prompt and run the following command:
kubectl port-forward svc/argocd-server -n argocd 8080:443
http://localhost:8080 in a web browser and log in as
admin using the initial password from the secret (or the updated password if changed via the CLI), then verify that the dashboard is accessible.
Deploying the Tekton CI pipeline
Now we are ready to deploy the Tekton CI pipeline for the sample application. Before you start, take a look at the Concepts page in the Tekton documentation to understand what the building blocks are and how they interact with one another. But in essence, we need to know the following:
A pipeline consists of tasks to perform the required build operations.
A task consists of steps to encapsulate a build operation, such as cloning a Git repository or updating Kubernetes YAML files in the configuration Git repository.
Tekton provides a catalog of shared tasks and pipelines that can be used out of the box.
A step is an operation run in a container in the Kubernetes cluster.
A volume-based workspace is used by tasks and steps to share data across the pipeline
Coming from a traditional CI/CD world of Jenkins and such, running a step in a container seems awfully inefficient to me. However it is not the case, since containers spin up relatively quickly and Tekton handles the lifecycle anyway.
The CI pipeline for our sample to-do list application consists of the following tasks:
Use the git-clone task from the Tekton catalog to clone the application repository into a (shared) workspace.
Use the kaniko task from the Tekton catalog to build both
todo-list-backendusing the provided Dockerfiles, and push the resulting images to the DigitalOcean Container Registry. Kaniko, by the way, is a tool to build container images inside a Kubernetes cluster.
Use a custom task we put together to clone the configuration repository, update the container image version in the Kubernetes YAML file, and commit the changes to Git (for ArgoCD to pick up).
Let's now set up the Tekton pipeline for the sample application.
Setting up the prerequisites
Perform the following steps to provision everything the pipeline needs in the Kubernetes cluster. There are many steps, so hang in there!
Create a namespace called
todo-list-cito host the Tekton pipeline resources with the following command:
kubectl create namespace todo-list-ci
Create a secret that the custom task uses to commit the updated Kubernetes YAML file to the configuration repository on GitHub. Since the step uses
gitcommand with no user interaction, I decided to use the
GIT_ASKPASSenvironment variable to provide the PAT after reading this article as a quick and dirty solution. Create a new YAML file using the following template (see also
tekton/config-git-askpass.yaml.templatein the sample code repository) and your GitHub PAT previously created:
apiVersion: v1 kind: Secret metadata: name: config-git-askpass type: Opaque stringData: .git-askpass: | echo "### Your GitHub PAT goes here ###"
Then create the secret with the following command:
kubectl apply -f config-git-askpass.yaml -n todo-list-ci
Create a secret that provides the
kanikotask a Docker config to push the application container images to our DigitalOcean Container Registry. Note that the
kanikotask expects the file name to be
config.json, which is different from the default file names that Kubernetes expects. In any case, create a new YAML file using the following template (see also
tekton/do-docker-config.yaml.templatein the sample code repository) and your DigitalOcean PAT previously created:
apiVersion: v1 kind: Secret metadata: name: do-docker-config type: kubernetes.io/dockercfgjson data: config.json: | ### Your DigitalOcean PAT goes here ###
Then create the secret with the following command:
kubectl apply -f do-docker-config.yaml -n todo-list-ci
kanikotasks from the Tekton catalog with the following commands:
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.5/git-clone.yaml -n todo-list-ci kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kaniko/0.5/kaniko.yaml -n todo-list-ci
The pipeline requires a shared workspace to store the source code and configuration from Git. In the
PipelineRunconfiguration, we need to provide a persistent volume claim (PVC) to provision the workspace. The YAML file is as follows (see also
tekton/todo-list-ci-pvc.yamlin the sample code repository):
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: todo-list-ci-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi
To create the PVC, run the following command:
kubectl apply -f todo-list-ci-pvc.yaml -n todo-list-ci
Installing the custom task resource
Since there is no readily available task to update the Kubernetes YAML file with new image versions, I had to create a custom task. Drawing inspiration from Sebastian Daschner's example, the resulting task resource is defined as follows (see also
tekton/update-config-task.yaml in the sample code repository):
apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: update-config spec: params: - name: buildVersion - name: gitUrl - name: gitBranch - name: k8sYaml - name: gitUserName - name: gitUserEmail workspaces: - name: git-source - name: git-askpass-secret steps: - name: git-checkout image: alpine/git:v2.26.2 workingDir: "$(workspaces.git-source.path)" script: | #!/usr/bin/env sh set -e cp $(workspaces.git-askpass-secret.path)/.git-askpass ~/ chmod +x ~/.git-askpass export GIT_ASKPASS=~/.git-askpass rm -rf config git clone -b $(inputs.params.gitBranch) $(inputs.params.gitUrl) config - name: update-yaml image: alpine/git:v2.26.2 workingDir: "$(workspaces.git-source.path)" script: | #!/usr/bin/env sh set -e cd config sed -i "s#/todo-list-frontend:[a-zA-Z0-9.]\\+#/todo-list-frontend:$(inputs.params.buildVersion)#" $(inputs.params.k8sYaml) sed -i "s#/todo-list-backend:[a-zA-Z0-9.]\\+#/todo-list-backend:$(inputs.params.buildVersion)#" $(inputs.params.k8sYaml) cat $(inputs.params.k8sYaml) - name: commit-push-changes-gitops image: alpine/git:v2.26.2 workingDir: "$(workspaces.git-source.path)" script: | #!/usr/bin/env sh set -e cd config cp $(workspaces.git-askpass-secret.path)/.git-askpass ~/ chmod +x ~/.git-askpass export GIT_ASKPASS=~/.git-askpass git config --global user.email "$(inputs.params.gitUserEmail)" git config --global user.name "$(inputs.params.gitUserName)" git add . git commit --allow-empty -m "[tekton] Set deployment to version $(inputs.params.buildVersion)" git push origin $(inputs.params.gitBranch)
As you can see, there is a common workspace used by all three steps, each of which runs a different container and script. The task is also using the secret that contains the GitHub PAT as a file that the
GIT_ASKPASS environment variable refers to. As for the logic, the task first clones the configuration repository on GitHub, then uses
sed to update the container image version in
todo-list.yaml, and finally commits/pushes the change into Git.
Now that you understand how this custom task works, create it in your Kubernetes cluster with the following command:
kubectl apply -f update-config-task.yaml -n todo-list-ci
Installing the pipeline resource
Lastly, we need to install the pipeline resource. Refer to the definition below (see also
tekton/todo-list-ci-pipeline.yaml in the sample code repository):
apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: todo-list-ci-pipeline spec: workspaces: - name: git-source - name: do-docker-config-secret - name: config-git-askpass-secret params: - name: buildVersion - name: appGitUrl - name: appGitBranch default: main - name: frontendPathToContext - name: frontendImageUrl - name: backendPathToContext - name: backendImageUrl - name: configGitUrl - name: configGitBranch default: main - name: k8sYaml - name: gitUserName - name: gitUserEmail tasks: - name: clone-repo taskRef: name: git-clone workspaces: - name: output workspace: git-source params: - name: url value: "$(params.appGitUrl)" - name: revision value: "$(params.appGitBranch)" - name: subdirectory value: "app" - name: deleteExisting value: "true" - name: frontend-build-image taskRef: name: kaniko runAfter: - clone-repo workspaces: - name: source workspace: git-source - name: dockerconfig workspace: docker-config-secret params: - name: CONTEXT value: app/$(params.frontendPathToContext) - name: IMAGE value: $(params.frontendImageUrl):$(params.buildVersion) - name: backend-build-image taskRef: name: kaniko runAfter: - clone-repo workspaces: - name: source workspace: git-source - name: dockerconfig workspace: do-docker-config-secret params: - name: CONTEXT value: app/$(params.backendPathToContext) - name: IMAGE value: $(params.backendImageUrl):$(params.buildVersion) - name: update-config taskRef: name: update-config runAfter: - frontend-build-image - backend-build-image workspaces: - name: git-source workspace: git-source - name: git-askpass-secret workspace: config-git-askpass-secret params: - name: buildVersion value: "$(params.buildVersion)" - name: gitUrl value: "$(params.configGitUrl)" - name: gitBranch value: "$(params.configGitBranch)" - name: k8sYaml value: "$(params.k8sYaml)" - name: gitUserName value: "$(params.gitUserName)" - name: gitUserEmail value: "$(params.gitUserEmail)"
The definition is a tad long, but it performs the tasks as explained earlier. Let's create it with the following command:
kubectl apply -f todo-list-ci-pipeline.yaml -n todo-list-ci
Now that our Tekton CI pipeline is ready to go, feel free to take a well-deserved break and digest all the information we've gone through so far.
Create an application in ArgoCD
At last, we need to create the CD pipeline to complete our solution. Luckily this is very simple to do in the ArgoCD UI. You could use a declarative setup to see full benefits, but let's take it easy for now. Follow the steps below to create the application:
Log in to the ArgoCD UI.
Click the New App button.
In the General section, enter an application name and select the
Scroll down to the Source section and enter your configuration repository URL and
todo-listfor the path.
Scroll down to the Destination section and select
https://kubernetes.default.svcas the cluster URL (this is the value for deploying to the same cluster in which ArgoCD is running) and enter
todo-listas the namespace. Click the Create button to create the app.
The todo-list app is now created with the missing and OutOfSync status:
Don't sync it just yet because we haven't deployed the container images to the container registry! We will get the container images ready by running the CI pipeline.
Running the Tekton CI pipeline
There are two ways to run a pipeline in Tekton:
Creating a PipelineRun Kubernetes custom resource
Configuring a Tekton Trigger to automatically start pipeline runs
Triggers can be overwhelming for beginners, so we will just manually create a PipelineRun resource to start the pipeline for this exercise. Create a new YAML file using the following template (see also
tekton/todo-list-ci-pipelinerun.yaml.template in the sample code repository) and replace the values with ones specific to your environment:
apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata: generateName: todo-list-ci- spec: pipelineRef: name: todo-list-ci-pipeline params: - name: buildVersion value: "0.0.1" - name: appGitUrl value: ### Your app Git repo URL goes here ### - name: appGitBranch value: ### Your app Git branch name goes here (typically main or master) ### - name: frontendPathToContext value: "todo-list-frontend" - name: frontendImageUrl value: "### Your DigitalOcean Container Registry URL goes here ###/todo-list-frontend" - name: backendPathToContext value: "todo-list-backend" - name: backendImageUrl value: ""### Your DigitalOcean Container Registry URL goes here ###/todo-list-backend" - name: configGitUrl value: ### Your config Git repo URL goes here ### - name: configGitBranch value: ### Your config Git branch name goes here (typically main or master) ### - name: k8sYaml value: "todo-list/todo-list.yaml" serviceAccountName: default workspaces: - name: git-source persistentVolumeClaim: claimName: todo-list-ci-pvc - name: do-docker-config-secret secret: secretName: do-docker-config - name: config-git-askpass-secret secret: secretName: config-git-askpass
To start a new pipeline run, run the following command:
kubectl create -f todo-list-ci-pipelinerun.yaml -n todo-list-ci
The command should complete with the PipelineRun ID similar to the following:
Now, head over to the Tekton Dashboard in a web browser. Select PipelineRuns from the left menu and you will see that our pipeline is running!
Click the PipelineRun name and you will see the progress. In the screenshot below, I opened the
build-and-push step under the
frontend-build-image task, so we can see the progress of Kaniko building the image for the frontend application.
When this is done, verify in your configuration GitHub repository that a commit has been made to update the container image versions to 0.0.1. Next we can verify the deployment in ArgoCD!
Synchronizing the application in ArgoCD for the first time
Head over to the ArgoCD UI in the web browser. Then follow these steps to manually synchronize the application manifests, which will trigger a full deployment. Click the Sync button in the todo-list application tile to open the synchronize dialog, then click the Synchronize button to start the process.
Wait for the status to show the green Healthy and Synced statuses, then click the tile to see the deployment details. In the application details page, you will see a visualization of the Kubernetes resources for the application and other details. To open the application, click the 3rd button inside the
todo-list ingress box as highlighted in the screenshot below:
Finally we get to see the fruit of our labor! Feel free to play around with the to-do list application to make sure that it is working.
Enabling auto-sync in ArgoCD to complete the CI/CD solution
As we have finally verified a deployment, we need to make one final change to complete our CI/CD solution. To truly enable GitOps and CD, we need to turn on auto-sync for the application in ArgoCD.
Back on the app details page in ArgoCD UI, click the App Details button to open the details dialog. In the Summary tab, scroll down to the Sync Policy section and click the Enable Auto-Sync button.
Then click the OK button when the prompt appears:
Lastly, we will trigger another run for the Tekton CI pipeline for the final verification. We could make some functional changes to the todo-list application, but to test auto-sync we can just update the image version to simulate an "application update". Edit the PipelineRun YAML file created earlier and change the
buildVersion parameter value to, say, 0.0.2. Then run the following command to start a new pipeline run:
kubectl create -f todo-list-ci-pipelinerun.yaml -n todo-list-ci
Check the pipeline progress in Tekton Dashboard. When the run completes, go back to ArgoCD UI and verify that it has automatically synchronize after on the Git changes. You may need to refresh the status in the UI to reflect the latest status. In the application details page, the current sync status now shows that version 0.0.2 is deployed:
Congratulations! You have successfully created a basic Kubernetes-native CI/CD solution using Tekton and ArgoCD on DigitalOcean!
After you have admired your creation and played with the environment more, don't forget to destroy the Kubernetes cluster in DigitalOcean and any leftover resources (volumes and load balancers for the ingress) to avoid unnecessary cost!
In this challenge, I have only scratched the surface of Tekton and ArgoCD. With my remaining DigitalOcean credits available for another month, I would like to build upon the current solution with the following:
Implement trigger to truly complete the CI pipeline
Extend the current solution to multiple environments (dev, QA, prod) and evaluate various high availability features
Research on best practices and improve the current solution
Check back later for new posts as I explore further!
During the challenge, I found the following resources that helped me understand how to use the tools and solve unexpected issues. Credit goes to the authors and I hope this curated list helps you as well.
Kubernetes-Native Build & Release Pipelines with Tekton and ArgoCD by Adri Villela - A comprehensive series on end-to-end Kubernetes-native CI/CD, as recommended on the challenge page itself.
Tekton with ArgoCD by Sebastian Daschner - A great blueprint for CI/CD with Tekton and ArgoCD. It saved me time scripting the Git update part for this challenge.
ArgoCD Tutorial for Beginners | GitOps CD for Kubernetes by TechWorld with Nana - A well-explained video tutorial for ArgoCD on YouTube. She has a knack for explaining complex concepts and I am a big fan of her videos.
How to fix in Kubernetes – Deleting a PVC stuck in status "Terminating" - This article helped me fix a problem where I could not delete a PVC and the associated volume.
The Vanilla DevOps Git Credentials & Private Packages Cheatsheet by AJ ONeal - This cheatsheet helped me choose the best Git credentials management strategy for this challenge.