Search for a command to run...
Learn how to create a basic Kubernetes-Native CI/CD solution using Tekton and ArgoCD on DigitalOcean.
No comments yet. Be the first to comment.
Learn how to deploy a self-hosted AI chat platform on AWS using LibreChat, MongoDB Atlas, Amazon ECS, and Amazon Bedrock, provisioned via Terraform.
Explore AWS Control Tower proactive controls, why they don’t work with Terraform, and a proof of concept that attempts to make them work. Introduction As a Terraform advocate and an AWS consultant who
I recently passed the Terraform Authoring and Operations Professional Exam. Here are my practical tips to help you prepare more effectively.
Learn how to monitor data ingestion jobs for Amazon Bedrock Knowledge Bases and enable notifications using CloudWatch Logs.
Learn how to enable logging for Amazon Bedrock Knowledge Bases, which uses the log delivery feature of Amazon CloudWatch Logs, with Terraform.
Having learned about the DigitalOcean Kubernetes Challenge, I decided to take it on to upskill myself on trending DevOps tools for Kubernetes, and to evaluate DigitalOcean's container services. For the challenge, I implemented a Kubernetes-native CI/CD solution using Tekton build and ArgoCD for deployment. In this article, I will share my experience and sample code I've put together, so you too can try it out.
You can find the sample code for the challenge at https://github.com/acwwat/do-k8s-gitops. The folder structure is as follows:
The app folder contains the source code for the sample to-do list application that we will build into Docker images and deployed to Kubernetes. The frontend todo-list-frontend is a basic Vue.js application and the backend todo-list-backend is a basic Java Spring Boot application with an embedded H2 database.
The config folder contains the starter Kubernetes YAML file template which we will use to deploy the application to the Kubernetes cluster.
The tekton folder contains the Tekton Kubernetes YAML files that defines the various components that form the CI pipeline.
Clone or download this GitHub repository as we will need it to set up the CI/CD solution.
You need to create two repositories - one for the application to drive CI and the other for the Kubernetes configuration to drive CD. I used GitHub myself, but you can also use other Git repositories so long as they are accessible by Tekton and ArgoCD.
For the application repository, check in the content of the app folder from the sample code repository.
For the configuration repository, check in the content of the config folder. Since the Kubernetes YAML file requires a hardcoded container image path from your DigitalOcean Container Registry, you need to copy todo-list/todo-list.yaml.template as todo-list/todo-list.yaml and replace the URL of your container registry once you have created it.
You need to create the following access tokens for integration purposes:
A GitHub personal access token for the Tekton pipeline to update the Kubernetes YAML files in the configuration repository. The PAT must have all scopes under the repo category selected.
A DigitalOcean personal access token for the Tekton pipeline to push the application Docker images to the DigitalOcean Container Registry. The PAT must have the write scope selected.
Copy the tokens and set them aside for now, as you will later store them as Kubernetes secrets.
To keep things simple, I chose to favor GUIs over CLI tools for this challenge. However you still need to install a few CLI tools as follows:
Setting up a Kubernetes cluster can be very involved, but DigitalOcean has made it simple. Once you have your account and logged into the dashboard, create a project and then create a Kubernetes cluster. For the challenge, I used the following settings:
After the Kubernetes cluster is provisioned after several minutes, install the NGINX Ingress Controller as a 1-Click App on the cluster details page:
An ingress controller is not a must, however I'd like to evaluate it on DigitalOcean for practical purposes. Allow it a few minutes to deploy. When it is done, run the doctl command from the cluster details page to connect kubectl to your cluster:
You need a container registry to store the application container images, so this is our chance to also evaluate the DigitalOcean Container Registry. In the DigitalOcean Dashboard, create a registry with the Basic subscription plan, as we require two repositories (one per Docker image) in the registry. When the registry is created, enable integration to your Kubernetes cluster in the settings.
Don't forget to create the Kubernetes YAML file in the configuration Git repository as mentioned earlier!
Tekton is an open-source, Kubernetes-native CI/CD framework. Tekton itself is a Kubernetes application and it provides custom resources as building blocks for pipelines. It took me a while to get past the finer granularity vs. GitHub Actions and Azure Pipelines, but I also appreciate its flexibility and tight integration with Kubernetes. In this challenge, I chose to implement CD using ArgoCD, however using Tekton for CD is also perfectly doable.
Installing Tekton is relatively simple. Follow the Getting Started guide to install the core components and set up the ConfigMaps for persistent volumes. You can also install the Tekton CLI and run the sample task if you like, but we won't use the CLI in this exercise. By default, Tekton uses the default service account in the cluster to run pipelines. While it would be a security best practice to create a separate service account for CI/CD, we will just keep the default for now.
As well, install the Tekton Dashboard which we will use to view pipeline progress. To not expose the Tekton Dashboard to the internet, we can use the port forwarding option to map a local port to the dashboard HTTP port. Open a new command prompt and run the following command:
kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097
Then open http://localhost:9097 in a web browser and verify that the dashboard is accessible.
Now let's switch gear to ArgoCD. ArgoCD is a declarative, GitOps CD tool for Kubernetes. Its concept is simple - detect changes between current state in the Kubernetes cluster and desired state per configuration in Git, and apply them either on-demand or automatically. With configurations in Git, various DevOps best practices such as version control, code review, and CD are possible.
Installing ArgoCD is also very simple - just follow the Getting Started guide. We won't need the CLI for the exercise, but feel free to install it when going through the guide. Once installation completes, you can forward the server port to access the ArgoCD UI without exposing it to the internet. Open a new command prompt and run the following command:
kubectl port-forward svc/argocd-server -n argocd 8080:443
Then open http://localhost:8080 in a web browser and log in as admin using the initial password from the secret (or the updated password if changed via the CLI), then verify that the dashboard is accessible.
Now we are ready to deploy the Tekton CI pipeline for the sample application. Before you start, take a look at the Concepts page in the Tekton documentation to understand what the building blocks are and how they interact with one another. But in essence, we need to know the following:
A pipeline consists of tasks to perform the required build operations.
A task consists of steps to encapsulate a build operation, such as cloning a Git repository or updating Kubernetes YAML files in the configuration Git repository.
Tekton provides a catalog of shared tasks and pipelines that can be used out of the box.
A step is an operation run in a container in the Kubernetes cluster.
A volume-based workspace is used by tasks and steps to share data across the pipeline
Coming from a traditional CI/CD world of Jenkins and such, running a step in a container seems awfully inefficient to me. However it is not the case, since containers spin up relatively quickly and Tekton handles the lifecycle anyway.
The CI pipeline for our sample to-do list application consists of the following tasks:
Use the git-clone task from the Tekton catalog to clone the application repository into a (shared) workspace.
Use the kaniko task from the Tekton catalog to build both todo-list-frontend and todo-list-backend using the provided Dockerfiles, and push the resulting images to the DigitalOcean Container Registry. Kaniko, by the way, is a tool to build container images inside a Kubernetes cluster.
Use a custom task we put together to clone the configuration repository, update the container image version in the Kubernetes YAML file, and commit the changes to Git (for ArgoCD to pick up).
Let's now set up the Tekton pipeline for the sample application.
Perform the following steps to provision everything the pipeline needs in the Kubernetes cluster. There are many steps, so hang in there!
Create a namespace called todo-list-ci to host the Tekton pipeline resources with the following command:
kubectl create namespace todo-list-ci
Create a secret that the custom task uses to commit the updated Kubernetes YAML file to the configuration repository on GitHub. Since the step uses git command with no user interaction, I decided to use the GIT_ASKPASS environment variable to provide the PAT after reading this article as a quick and dirty solution. Create a new YAML file using the following template (see also tekton/config-git-askpass.yaml.template in the sample code repository) and your GitHub PAT previously created:
apiVersion: v1
kind: Secret
metadata:
name: config-git-askpass
type: Opaque
stringData:
.git-askpass: |
echo "### Your GitHub PAT goes here ###"
Then create the secret with the following command:
kubectl apply -f config-git-askpass.yaml -n todo-list-ci
Create a secret that provides the kaniko task a Docker config to push the application container images to our DigitalOcean Container Registry. Note that the kaniko task expects the file name to be config.json, which is different from the default file names that Kubernetes expects. In any case, create a new YAML file using the following template (see also tekton/do-docker-config.yaml.template in the sample code repository) and your DigitalOcean PAT previously created:
apiVersion: v1
kind: Secret
metadata:
name: do-docker-config
type: kubernetes.io/dockercfgjson
data:
config.json: |
### Your DigitalOcean PAT goes here ###
Then create the secret with the following command:
kubectl apply -f do-docker-config.yaml -n todo-list-ci
Install the git-clone and kaniko tasks from the Tekton catalog with the following commands:
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.5/git-clone.yaml -n todo-list-ci
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kaniko/0.5/kaniko.yaml -n todo-list-ci
The pipeline requires a shared workspace to store the source code and configuration from Git. In the PipelineRun configuration, we need to provide a persistent volume claim (PVC) to provision the workspace. The YAML file is as follows (see also tekton/todo-list-ci-pvc.yaml in the sample code repository):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: todo-list-ci-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
To create the PVC, run the following command:
kubectl apply -f todo-list-ci-pvc.yaml -n todo-list-ci
Since there is no readily available task to update the Kubernetes YAML file with new image versions, I had to create a custom task. Drawing inspiration from Sebastian Daschner's example, the resulting task resource is defined as follows (see also tekton/update-config-task.yaml in the sample code repository):
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: update-config
spec:
params:
- name: buildVersion
- name: gitUrl
- name: gitBranch
- name: k8sYaml
- name: gitUserName
- name: gitUserEmail
workspaces:
- name: git-source
- name: git-askpass-secret
steps:
- name: git-checkout
image: alpine/git:v2.26.2
workingDir: "$(workspaces.git-source.path)"
script: |
#!/usr/bin/env sh
set -e
cp $(workspaces.git-askpass-secret.path)/.git-askpass ~/
chmod +x ~/.git-askpass
export GIT_ASKPASS=~/.git-askpass
rm -rf config
git clone -b $(inputs.params.gitBranch) $(inputs.params.gitUrl) config
- name: update-yaml
image: alpine/git:v2.26.2
workingDir: "$(workspaces.git-source.path)"
script: |
#!/usr/bin/env sh
set -e
cd config
sed -i "s#/todo-list-frontend:[a-zA-Z0-9.]\\+#/todo-list-frontend:$(inputs.params.buildVersion)#" $(inputs.params.k8sYaml)
sed -i "s#/todo-list-backend:[a-zA-Z0-9.]\\+#/todo-list-backend:$(inputs.params.buildVersion)#" $(inputs.params.k8sYaml)
cat $(inputs.params.k8sYaml)
- name: commit-push-changes-gitops
image: alpine/git:v2.26.2
workingDir: "$(workspaces.git-source.path)"
script: |
#!/usr/bin/env sh
set -e
cd config
cp $(workspaces.git-askpass-secret.path)/.git-askpass ~/
chmod +x ~/.git-askpass
export GIT_ASKPASS=~/.git-askpass
git config --global user.email "$(inputs.params.gitUserEmail)"
git config --global user.name "$(inputs.params.gitUserName)"
git add .
git commit --allow-empty -m "[tekton] Set deployment to version $(inputs.params.buildVersion)"
git push origin $(inputs.params.gitBranch)
As you can see, there is a common workspace used by all three steps, each of which runs a different container and script. The task is also using the secret that contains the GitHub PAT as a file that the GIT_ASKPASS environment variable refers to. As for the logic, the task first clones the configuration repository on GitHub, then uses sed to update the container image version in todo-list.yaml, and finally commits/pushes the change into Git.
Now that you understand how this custom task works, create it in your Kubernetes cluster with the following command:
kubectl apply -f update-config-task.yaml -n todo-list-ci
Lastly, we need to install the pipeline resource. Refer to the definition below (see also tekton/todo-list-ci-pipeline.yaml in the sample code repository):
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: todo-list-ci-pipeline
spec:
workspaces:
- name: git-source
- name: do-docker-config-secret
- name: config-git-askpass-secret
params:
- name: buildVersion
- name: appGitUrl
- name: appGitBranch
default: main
- name: frontendPathToContext
- name: frontendImageUrl
- name: backendPathToContext
- name: backendImageUrl
- name: configGitUrl
- name: configGitBranch
default: main
- name: k8sYaml
- name: gitUserName
- name: gitUserEmail
tasks:
- name: clone-repo
taskRef:
name: git-clone
workspaces:
- name: output
workspace: git-source
params:
- name: url
value: "$(params.appGitUrl)"
- name: revision
value: "$(params.appGitBranch)"
- name: subdirectory
value: "app"
- name: deleteExisting
value: "true"
- name: frontend-build-image
taskRef:
name: kaniko
runAfter:
- clone-repo
workspaces:
- name: source
workspace: git-source
- name: dockerconfig
workspace: docker-config-secret
params:
- name: CONTEXT
value: app/$(params.frontendPathToContext)
- name: IMAGE
value: $(params.frontendImageUrl):$(params.buildVersion)
- name: backend-build-image
taskRef:
name: kaniko
runAfter:
- clone-repo
workspaces:
- name: source
workspace: git-source
- name: dockerconfig
workspace: do-docker-config-secret
params:
- name: CONTEXT
value: app/$(params.backendPathToContext)
- name: IMAGE
value: $(params.backendImageUrl):$(params.buildVersion)
- name: update-config
taskRef:
name: update-config
runAfter:
- frontend-build-image
- backend-build-image
workspaces:
- name: git-source
workspace: git-source
- name: git-askpass-secret
workspace: config-git-askpass-secret
params:
- name: buildVersion
value: "$(params.buildVersion)"
- name: gitUrl
value: "$(params.configGitUrl)"
- name: gitBranch
value: "$(params.configGitBranch)"
- name: k8sYaml
value: "$(params.k8sYaml)"
- name: gitUserName
value: "$(params.gitUserName)"
- name: gitUserEmail
value: "$(params.gitUserEmail)"
The definition is a tad long, but it performs the tasks as explained earlier. Let's create it with the following command:
kubectl apply -f todo-list-ci-pipeline.yaml -n todo-list-ci
Now that our Tekton CI pipeline is ready to go, feel free to take a well-deserved break and digest all the information we've gone through so far.
At last, we need to create the CD pipeline to complete our solution. Luckily this is very simple to do in the ArgoCD UI. You could use a declarative setup to see full benefits, but let's take it easy for now. Follow the steps below to create the application:
Log in to the ArgoCD UI.
Click the New App button.
In the General section, enter an application name and select the default project.
Scroll down to the Source section and enter your configuration repository URL and todo-list for the path.
Scroll down to the Destination section and select https://kubernetes.default.svc as the cluster URL (this is the value for deploying to the same cluster in which ArgoCD is running) and enter todo-list as the namespace. Click the Create button to create the app.
The todo-list app is now created with the missing and OutOfSync status:
Don't sync it just yet because we haven't deployed the container images to the container registry! We will get the container images ready by running the CI pipeline.
There are two ways to run a pipeline in Tekton:
Creating a PipelineRun Kubernetes custom resource
Configuring a Tekton Trigger to automatically start pipeline runs
Triggers can be overwhelming for beginners, so we will just manually create a PipelineRun resource to start the pipeline for this exercise. Create a new YAML file using the following template (see also tekton/todo-list-ci-pipelinerun.yaml.template in the sample code repository) and replace the values with ones specific to your environment:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: todo-list-ci-
spec:
pipelineRef:
name: todo-list-ci-pipeline
params:
- name: buildVersion
value: "0.0.1"
- name: appGitUrl
value: ### Your app Git repo URL goes here ###
- name: appGitBranch
value: ### Your app Git branch name goes here (typically main or master) ###
- name: frontendPathToContext
value: "todo-list-frontend"
- name: frontendImageUrl
value: "### Your DigitalOcean Container Registry URL goes here ###/todo-list-frontend"
- name: backendPathToContext
value: "todo-list-backend"
- name: backendImageUrl
value: ""### Your DigitalOcean Container Registry URL goes here ###/todo-list-backend"
- name: configGitUrl
value: ### Your config Git repo URL goes here ###
- name: configGitBranch
value: ### Your config Git branch name goes here (typically main or master) ###
- name: k8sYaml
value: "todo-list/todo-list.yaml"
serviceAccountName: default
workspaces:
- name: git-source
persistentVolumeClaim:
claimName: todo-list-ci-pvc
- name: do-docker-config-secret
secret:
secretName: do-docker-config
- name: config-git-askpass-secret
secret:
secretName: config-git-askpass
To start a new pipeline run, run the following command:
kubectl create -f todo-list-ci-pipelinerun.yaml -n todo-list-ci
The command should complete with the PipelineRun ID similar to the following:
pipelinerun.tekton.dev/todo-list-ci-r2f94 created
Now, head over to the Tekton Dashboard in a web browser. Select PipelineRuns from the left menu and you will see that our pipeline is running!
Click the PipelineRun name and you will see the progress. In the screenshot below, I opened the build-and-push step under the frontend-build-image task, so we can see the progress of Kaniko building the image for the frontend application.
When this is done, verify in your configuration GitHub repository that a commit has been made to update the container image versions to 0.0.1. Next we can verify the deployment in ArgoCD!
Head over to the ArgoCD UI in the web browser. Then follow these steps to manually synchronize the application manifests, which will trigger a full deployment. Click the Sync button in the todo-list application tile to open the synchronize dialog, then click the Synchronize button to start the process.
Wait for the status to show the green Healthy and Synced statuses, then click the tile to see the deployment details. In the application details page, you will see a visualization of the Kubernetes resources for the application and other details. To open the application, click the 3rd button inside the todo-list ingress box as highlighted in the screenshot below:
Finally we get to see the fruit of our labor! Feel free to play around with the to-do list application to make sure that it is working.
As we have finally verified a deployment, we need to make one final change to complete our CI/CD solution. To truly enable GitOps and CD, we need to turn on auto-sync for the application in ArgoCD.
Back on the app details page in ArgoCD UI, click the App Details button to open the details dialog. In the Summary tab, scroll down to the Sync Policy section and click the Enable Auto-Sync button.
Then click the OK button when the prompt appears:
Lastly, we will trigger another run for the Tekton CI pipeline for the final verification. We could make some functional changes to the todo-list application, but to test auto-sync we can just update the image version to simulate an "application update". Edit the PipelineRun YAML file created earlier and change the buildVersion parameter value to, say, 0.0.2. Then run the following command to start a new pipeline run:
kubectl create -f todo-list-ci-pipelinerun.yaml -n todo-list-ci
Check the pipeline progress in Tekton Dashboard. When the run completes, go back to ArgoCD UI and verify that it has automatically synchronize after on the Git changes. You may need to refresh the status in the UI to reflect the latest status. In the application details page, the current sync status now shows that version 0.0.2 is deployed:
Congratulations! You have successfully created a basic Kubernetes-native CI/CD solution using Tekton and ArgoCD on DigitalOcean!
After you have admired your creation and played with the environment more, don't forget to destroy the Kubernetes cluster in DigitalOcean and any leftover resources (volumes and load balancers for the ingress) to avoid unnecessary cost!
In this challenge, I have only scratched the surface of Tekton and ArgoCD. With my remaining DigitalOcean credits available for another month, I would like to build upon the current solution with the following:
Implement trigger to truly complete the CI pipeline
Extend the current solution to multiple environments (dev, QA, prod) and evaluate various high availability features
Research on best practices and improve the current solution
Check back later for new posts as I explore further!
During the challenge, I found the following resources that helped me understand how to use the tools and solve unexpected issues. Credit goes to the authors and I hope this curated list helps you as well.
Kubernetes-Native Build & Release Pipelines with Tekton and ArgoCD by Adri Villela - A comprehensive series on end-to-end Kubernetes-native CI/CD, as recommended on the challenge page itself.
Tekton with ArgoCD by Sebastian Daschner - A great blueprint for CI/CD with Tekton and ArgoCD. It saved me time scripting the Git update part for this challenge.
ArgoCD Tutorial for Beginners | GitOps CD for Kubernetes by TechWorld with Nana - A well-explained video tutorial for ArgoCD on YouTube. She has a knack for explaining complex concepts and I am a big fan of her videos.
How to fix in Kubernetes – Deleting a PVC stuck in status "Terminating" - This article helped me fix a problem where I could not delete a PVC and the associated volume.
The Vanilla DevOps Git Credentials & Private Packages Cheatsheet by AJ ONeal - This cheatsheet helped me choose the best Git credentials management strategy for this challenge.