CICD jenkins pipeline triggered by github webhook
In this tutorial I want to demonstrate how to build a jenkins cicd pipeline for a simple flask application.
The pipeline will be triggered by a webhook on every push to main branch in condition that the commit message contains the following pattern “version X.X.X”
Adittionally, the GitHub webhook payload will be validated with a secret.
The pipeline steps will be:
1. Checkout the code.
2. Verify commit message.
3. Build a docker image.
4. Run some tests.
5. Push the docker image to docker hub.
6. Update the image tag on HELM github repo.
7. Post stage to clean the area.
This is my app repository, it’s a flask app that logs medication consumption, this app will be the example app for CICD pipeline.
https://github.com/EliBukin/button-logger-app
if you need an idea of how to deploy jenkins on kubernetes you can find it here.
and here you could find instructions for deploying and configuring ArgoCD on kubernetes.
What is Jenkins?
Jenkins is an open-source automation server widely used for continuous integration (CI) and continuous delivery (CD). It helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration and delivery. Here are some key features and components of Jenkins:
Key Features
1. Extensible and Flexible:
– Jenkins has a vast ecosystem of plugins that extend its capabilities to support various technologies and tools used in the development process.
2. Distributed Builds:
– Jenkins can distribute build tasks across multiple machines, helping to optimize resource usage and speed up the build process.
3. Pipeline as Code:
– Jenkins supports writing build processes as code using its Pipeline DSL (Domain Specific Language), allowing users to version control their build configurations and define complex workflows.
4. Integration with Various Tools:
– Jenkins integrates with numerous tools and technologies, including version control systems (e.g., Git), build tools (e.g., Maven, Gradle), testing frameworks, and deployment tools.
5. Web-based Interface:
– Jenkins provides an easy-to-use web interface that allows users to configure and monitor builds and deployments.
Components
1. Master:
– The Jenkins master is the central server that manages and schedules build jobs, maintains configurations, and provides the user interface.
2. Agents/Slaves:
– Jenkins agents (also known as slaves) are machines that run build tasks delegated by the master. Agents can run on different platforms and help distribute the workload.
3. Jobs/Projects:
– In Jenkins, a job or project represents a runnable task. This could be building software, running tests, or deploying an application. Users can configure jobs to be triggered manually or automatically.
4. Pipelines:
– Pipelines define the sequence of stages and steps required to build, test, and deploy an application. They can be written as code using Jenkins Pipeline DSL and stored in version control systems.
Use Cases
1. Continuous Integration:
– Automate the integration of code changes from multiple contributors into a shared repository, ensuring that new code integrates smoothly with existing code.
2. Continuous Delivery:
– Automate the deployment of code to production environments, ensuring that applications can be reliably released at any time.
3. Automated Testing:
– Run automated tests as part of the build process to identify issues early in the development cycle.
4. Monitoring and Reporting:
– Generate reports and monitor the status of builds and deployments, helping teams track progress and identify issues.
Overall, Jenkins is a powerful tool that enhances the efficiency and reliability of software development and delivery processes through automation.
Let’s start with creating a webhook that will trigger our Jenkins CICD pipeline every time new code pushed to master.
Navigate to GitHub repo settings –> webhooks –> and hit the Add webhook button.
It’s quite straight forward, not much to mess with, type your jenkins URL.
On the content type field select “application/json”
Secret, type a random string of characters, I think 20 will be enough, but don’t forget the string yet, you will need to it configure a credentials object in Jenkins.
NOTE: I strongly advice to create a webhook secret, to avoid malicious pipeline triggering.
this secret purpose is to validate the authenticity of the webhook payload.
The next step is to create a Personal Access Token PAT, later we will configure Jenkins to use it for GitHub authentication.
On the right side of your GitHub console click on settings and navigate to “Developer Settings”, there you will see “Tokens”, click it and configure a PAT.
copy the key for later.
Now let’s configure that PAT as a Jenkins credentials object.
Login to Jenkins –> Manage Jenkins –> Credentials –> System –> Global credentials (unrestricted)
Click the Add Credentials button and configure the PAT you got from GitHub as a secret text.
Another credentials object that has to be created is the webhook secret, remember the 20 character string you used in the webhook configuration? Well, now we set it in Jenkins.
Dashboard –> Manage Jenkins –> Credentials –> System –> Global credentials (unrestricted)
Click the Add Credentials button and configure the 20 character string as a secret text.
One more set of credentials we will have to define for our container registry authentication, in my case it is Docker Hub.
Dashboard –> Manage Jenkins –> Credentials –> System –> Global credentials (unrestricted)
Click the Add Credentials button and configure credentials for container registry.
Now we can proceed to the pipeline building.
Navigate to dashboard –> new item –> pipeline
In the Build Triggers section check the “GitHub hook trigger for GITScm polling” checkbox.
Select Pipeline script and start writing into that tiny little window that I dunno how Jenkins dev team didn’t change to something more appropriate.
The first part of the pipeline will be agent, trigger, environment variables and the checkout stage.
As you can see we are running on any available agent, we have our environment variables that we will need down the rabbit hole, and the first stage which is the checkout stage that checks out the application repository.
pipeline {
agent any
triggers {
githubPush()
}
environment {
VERSION = 'unset'
DOCKER_IMAGE = 'elibukin/button-logger-app'
HELM_REPO_URL = 'https://github.com/EliBukin/helm-charts-for-button-logger-app.git'
HELM_REPO_BRANCH = 'main'
}
stages {
stage('Checkout') {
steps {
script {
withCredentials([string(credentialsId: 'github_pat', variable: 'GITHUB_TOKEN')]) {
git url: "https://${GITHUB_TOKEN}@github.com/EliBukin/button-logger-app.git",
branch: 'main'
}
}
}
}
The next stage will be the “Check commit message and extrace version” stage.
This stage is looking for a pattern (version X.X.X) in the commit message, if it is absent then the pipeline will abort.
The pattern “version X.X.X” is needed in the commit message cuz that version will be the tag of the new container build by the pipeline.
stage('Check Commit Message and Extract Version') {
steps {
script {
def commitMessage = sh(script: "git log -1 --pretty=%B", returnStdout: true).trim()
echo "Commit message: ${commitMessage}"
def matcher = (commitMessage =~ /version (\d+\.\d+\.\d+)/)
if (matcher) {
VERSION = matcher[0][1]
env.VERSION = VERSION
echo "Extracted version: ${VERSION}"
} else {
error("Build aborted: Commit message does not contain version pattern.")
}
if (VERSION == 'unset' || VERSION == null) {
error("VERSION is not set properly. Current value: ${VERSION}")
}
}
}
}The next two stages are pretty trivial, build the docker image and run some tests on the app running from that image.
stage('Build Docker Image') {
steps {
script {
echo "Building Docker image with tag: ${DOCKER_IMAGE}:${VERSION}"
sh "docker build -t ${DOCKER_IMAGE}:${VERSION} ./app"
}
}
}
stage('Run Tests in Container') {
steps {
script {
echo "Running tests in container: ${DOCKER_IMAGE}:${VERSION}"
sh "docker run --rm ${DOCKER_IMAGE}:${VERSION} python -m unittest discover /app/tests"
}
}
}
Next stage will be the stage where we push the docker image to a registry, in this case Docker Hub.
I am using credentials from Jenkins internal credential objects, the credentials object is “dockerhub” which contains the user and the password for the registry.
stage('Push Docker Image') {
steps {
script {
echo "Pushing Docker image: ${DOCKER_IMAGE}:${VERSION}"
withCredentials([usernamePassword(credentialsId: 'dockerhub', usernameVariable: 'DOCKER_USERNAME', passwordVariable: 'DOCKER_PASSWORD')]) {
sh 'echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin'
sh "docker push ${DOCKER_IMAGE}:${VERSION}"
}
}
}
}The next stage will be the stage that updates the HELM chart repository, the application is deployed with ArgoCD so we change the tags on the helm charts repo.
The stage clones the repo and changes the tags in velues.yaml and chart.yaml files.
stage('Update Helm Charts') {
steps {
script {
withCredentials([string(credentialsId: 'github_pat', variable: 'GITHUB_TOKEN')]) {
// Clone the Helm charts repository
sh "git clone https://${GITHUB_TOKEN}@${HELM_REPO_URL.replaceFirst('https://', '')} helm-charts"
dir('helm-charts') {
// Update the image tag in values.yaml
sh "sed -i 's/tag: \".*\"/tag: \"${VERSION}\"/' button-logger-app/values.yaml"
// Update appVersion in Chart.yaml
sh "sed -i 's/appVersion: \".*\"/appVersion: \"${VERSION}\"/' button-logger-app/Chart.yaml"
// Commit and push changes
sh "git config user.email '<your-email>'"
sh "git config user.name '<your-username>'"
sh "git add ."
sh "git commit -m 'Update app version to ${VERSION}'"
sh "git push origin ${HELM_REPO_BRANCH}"
}
}
}
}
}
}And the last stage is the cleanup stage, where we clean after us.
post {
always {
script {
echo "Cleaning up..."
echo "VERSION: ${VERSION}"
echo "DOCKER_IMAGE: ${DOCKER_IMAGE}"
if (VERSION != null && VERSION != 'unset') {
sh "docker rmi ${DOCKER_IMAGE}:${VERSION} || true"
} else {
echo "Skipping image removal because VERSION is not set properly"
}
cleanWs()
}
}
failure {
echo "Pipeline failed. Please check the logs for details."
}
}
}Here is the full file:
pipeline {
agent any
triggers {
githubPush()
}
environment {
VERSION = 'unset'
DOCKER_IMAGE = 'elibukin/button-logger-app'
HELM_REPO_URL = 'https://github.com/EliBukin/helm-charts-for-button-logger-app.git'
HELM_REPO_BRANCH = 'main'
}
stages {
stage('Checkout') {
steps {
script {
withCredentials([string(credentialsId: 'github_pat', variable: 'GITHUB_TOKEN')]) {
git url: "https://${GITHUB_TOKEN}@github.com/EliBukin/button-logger-app.git",
branch: 'main'
}
}
}
}
stage('Check Commit Message and Extract Version') {
steps {
script {
def commitMessage = sh(script: "git log -1 --pretty=%B", returnStdout: true).trim()
echo "Commit message: ${commitMessage}"
def matcher = (commitMessage =~ /version (\d+\.\d+\.\d+)/)
if (matcher) {
VERSION = matcher[0][1]
env.VERSION = VERSION
echo "Extracted version: ${VERSION}"
} else {
error("Build aborted: Commit message does not contain version pattern.")
}
if (VERSION == 'unset' || VERSION == null) {
error("VERSION is not set properly. Current value: ${VERSION}")
}
}
}
}
stage('Build Docker Image') {
steps {
script {
echo "Building Docker image with tag: ${DOCKER_IMAGE}:${VERSION}"
sh "docker build -t ${DOCKER_IMAGE}:${VERSION} ./app"
}
}
}
stage('Run Tests in Container') {
steps {
script {
echo "Running tests in container: ${DOCKER_IMAGE}:${VERSION}"
sh "docker run --rm ${DOCKER_IMAGE}:${VERSION} python -m unittest discover /app/tests"
}
}
}
stage('Push Docker Image') {
steps {
script {
echo "Pushing Docker image: ${DOCKER_IMAGE}:${VERSION}"
withCredentials([usernamePassword(credentialsId: 'dockerhub', usernameVariable: 'DOCKER_USERNAME', passwordVariable: 'DOCKER_PASSWORD')]) {
sh 'echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin'
sh "docker push ${DOCKER_IMAGE}:${VERSION}"
}
}
}
}
stage('Update Helm Charts') {
steps {
script {
withCredentials([string(credentialsId: 'github_pat', variable: 'GITHUB_TOKEN')]) {
// Clone the Helm charts repository
sh "git clone https://${GITHUB_TOKEN}@${HELM_REPO_URL.replaceFirst('https://', '')} helm-charts"
dir('helm-charts') {
// Update the image tag in values.yaml
sh "sed -i 's/tag: \".*\"/tag: \"${VERSION}\"/' button-logger-app/values.yaml"
// Update appVersion in Chart.yaml
sh "sed -i 's/appVersion: \".*\"/appVersion: \"${VERSION}\"/' button-logger-app/Chart.yaml"
// Commit and push changes
sh "git config user.email '<github-email>'"
sh "git config user.name '<github-username>'"
sh "git add ."
sh "git commit -m 'Update app version to ${VERSION}'"
sh "git push origin ${HELM_REPO_BRANCH}"
}
}
}
}
}
}
post {
always {
script {
echo "Cleaning up..."
echo "VERSION: ${VERSION}"
echo "DOCKER_IMAGE: ${DOCKER_IMAGE}"
if (VERSION != null && VERSION != 'unset') {
sh "docker rmi ${DOCKER_IMAGE}:${VERSION} || true"
} else {
echo "Skipping image removal because VERSION is not set properly"
}
cleanWs()
}
}
failure {
echo "Pipeline failed. Please check the logs for details."
}
}
}
The next thing to do is to configure a GitHub server on Jenkins.
Navigate to Dashboard –> Manage Jenkins –> System –> GitHub
Add a GitHub server and set credentials for GitHub, this is the PAT, that will be used to authenticate to GitHub to manipulate the HELM repo.
Check “Manage Hooks” box.
Additionally we must set a secret for GitHub webhook, that will validate webhook payload.
Navigate to Dashboard –> Manage Jenkins –> System –> GitHub –Advanced
Add a GitHub server and set shared secret credentials for GitHub webhook.
Now we can finally test the pipeline, there is a thing to take into consideration, you got to run the pipeline once manually, it won’t work otherwise, it’s a thing with Jenkins.