Separating CI and CD into Independent Jenkins Pipelines

Everything we built so far lives in one pipeline: clone, build, image, push, deploy. For a small project that is perfectly fine. As projects and teams grow, one big pipeline starts to hurt:

• You cannot redeploy last week's version without rebuilding it from old code.
• A change to the deployment steps means editing the same job that builds the code.
• Audits get harder: "who deployed what, and when?" is mixed in with hundreds of build logs.
• Often different people own building (developers) and releasing (platform or operations team).

The usual answer is to split the work. CI (continuous integration) takes code and produces a tested, versioned artifact, here a Docker image. CD (continuous delivery or deployment) takes a version of that artifact and puts it into an environment. The two are separate Jenkins jobs, and CI starts CD automatically when it succeeds.

What the lecture builds

The video creates two Pipeline jobs:

The CD Trigger stage was generated with the Snippet Generator: Pipeline Syntax > build: Build a job, Project to Build CD Pipeline, Wait for completion and Propagate errors ticked. It produced:

build 'CD Pipeline'

Running only CI Pipeline started CD Pipeline by itself a few seconds later:

The CD Trigger stage stays blue (running) while it waits for the CD Pipeline to finish, because Wait for completion was ticked.

The gap in that version

Look closely at what the two jobs pass to each other: nothing. In the lecture's CI pipeline the image is built on the Jenkins server only, never pushed. The CD pipeline then applies k8s-deployment.yaml, which says image: haidertelusko/my-web-app:latest. The cluster pulls latest from Docker Hub, which is still the image pushed by the previous lesson's pipeline. So the new code never reaches the cluster, and because the manifest did not change, kubectl apply reports unchanged anyway.

Two fixes make the split work properly:

1. CI must push the image, with a unique tag. The cluster can only run images it can pull from a registry.
2. CI must tell CD which tag to deploy, through a build parameter.

Step 1: the CD pipeline

Create the CD job first, because the CI job needs to point at it. Use a name without spaces, for example cd-pipeline. Spaces work, but they turn into %20 in URLs and are easy to mistype in build job:.

pipeline {
    agent any

    parameters {
        string(name: 'IMAGE_TAG', defaultValue: '', description: 'Image tag to deploy, for example 42')
    }

    environment {
        IMAGE_NAME = 'your-dockerhub-username/my-web-app'
    }

    stages {
        stage('Validate') {
            steps {
                script {
                    if (!params.IMAGE_TAG?.trim()) {
                        error('IMAGE_TAG is empty. Start this job from the CI pipeline or enter a tag.')
                    }
                }
            }
        }

        stage('git clone') {
            steps {
                git branch: 'main', url: 'https://github.com/<your-github-user>/WebAppMaven.git'
            }
        }

        stage('k8s - deployment') {
            steps {
                sh 'sed -i "s|image: .*|image: $IMAGE_NAME:$IMAGE_TAG|" k8s-deployment.yaml'
                sh 'grep "image:" k8s-deployment.yaml'
                sh 'kubectl apply -f k8s-deployment.yaml'
                sh 'kubectl rollout status deployment/web-app-deployment --timeout=180s'
            }
        }
    }

    post {
        success {
            echo "Deployed ${IMAGE_NAME}:${params.IMAGE_TAG}"
        }
        failure {
            echo "Deployment of ${params.IMAGE_TAG} failed. The previous version is still serving traffic."
        }
    }
}

The lecture's CD job also had tools { maven 'maven' }. As the instructor says, it is not needed: this job never runs Maven.

Step 2: the CI pipeline

pipeline {
    agent any

    parameters {
        booleanParam(name: 'DEPLOY', defaultValue: true, description: 'Trigger the CD pipeline after a successful build')
    }

    environment {
        DOCKERHUB_USER = 'your-dockerhub-username'
        IMAGE_NAME     = "${DOCKERHUB_USER}/my-web-app"
        IMAGE_TAG      = "${BUILD_NUMBER}"
    }

    tools {
        maven 'maven'
    }

    stages {
        stage('git clone') {
            steps {
                git branch: 'main', url: 'https://github.com/<your-github-user>/WebAppMaven.git'
            }
        }

        stage('maven build') {
            steps {
                sh 'mvn clean package'
            }
        }

        stage('Build Docker Image') {
            steps {
                sh 'docker build -t $IMAGE_NAME:$IMAGE_TAG -t $IMAGE_NAME:latest .'
            }
        }

        stage('Docker Push') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'dockerhub-creds',
                                                  usernameVariable: 'DOCKER_USER',
                                                  passwordVariable: 'DOCKER_PASS')]) {
                    sh '''
                        echo "$DOCKER_PASS" | docker login -u "$DOCKER_USER" --password-stdin
                        docker push $IMAGE_NAME:$IMAGE_TAG
                        docker push $IMAGE_NAME:latest
                    '''
                }
            }
        }

        stage('CD Trigger') {
            when {
                expression { return params.DEPLOY }
            }
            steps {
                build job: 'cd-pipeline',
                      parameters: [string(name: 'IMAGE_TAG', value: env.IMAGE_TAG)],
                      wait: true,
                      propagate: true
            }
        }
    }

    post {
        always {
            sh 'docker logout || true'
            sh 'docker rmi $IMAGE_NAME:$IMAGE_TAG $IMAGE_NAME:latest || true'
        }
    }
}

Run ci-pipeline. The sequence is:

1. CI clones, builds, tests, builds my-web-app:<n> and pushes it.
2. The CD Trigger stage starts cd-pipeline with IMAGE_TAG=<n> and waits.
3. CD validates, clones the manifest, sets the image to <n>, applies it and waits for the rollout.
4. CD finishes, and CI finishes with the same result.

On the dashboard, open cd-pipeline: its build says Started by upstream project "ci-pipeline" build number n. That line is your audit trail, from deployment back to the exact CI build and commit.

Rolling back with the CD job

Here is the real benefit of the split. Suppose build 15 has a bug and 14 was fine. You do not need to revert code and wait for a full build. Open cd-pipeline, click Build with Parameters, enter 14, and run it. The CD job deploys the image that CI already built and pushed for build 14, in under a minute.

That only works because every build has its own tag on Docker Hub. With latest alone, version 14 would already be gone.

When to keep one pipeline

Splitting is not always better. For a personal project or a single small service, one pipeline is easier to read and maintain. Split when you need to redeploy old versions, promote the same image through several environments (dev, then staging, then production, each with its own CD job or stage), restrict who can deploy, or audit deployments separately from builds.

Common errors

Clean up

kubectl delete -f k8s-deployment.yaml
eksctl delete cluster --name telusko-learning --region ap-south-1 --wait

Try It Yourself

1. Create cd-pipeline and ci-pipeline as above and confirm that one click on CI deploys a new version.
2. Find the "Started by upstream project" line in the CD build and follow it back to the CI build.
3. Deploy an older tag by running cd-pipeline with Build with Parameters, and watch the rollout with kubectl get pods -w.
4. Make the CD job fail (enter a tag that does not exist) and see how propagate affects the CI result. Try propagate: false once.
5. Run CI with DEPLOY unticked and confirm the CD Trigger stage is skipped.

What you built across the bootcamp

Take a moment to look at what this last pipeline actually does. A git push starts a chain that touches almost every course in the bootcamp:

• Linux and Bash: every server here is Ubuntu, and you fixed users, groups, permissions and file ownership along the way (the jenkins user, the docker group, chmod 600 on the kubeconfig).
• Git and GitHub: the repository is the single source of truth for code, Dockerfile and manifests.
• Maven: compiled, tested and packaged the WAR in the CI stage.
• AWS: EC2 instances, an IAM role instead of access keys, a load balancer and a VPC created for the cluster.
• Terraform and Ansible: we clicked through the console to create these servers, but you now know how to create and configure exactly the same EC2 instances as code.
• Docker: turned the WAR into an image, tagged per build and stored on Docker Hub.
• Kubernetes on EKS: ran two replicas, exposed them through a LoadBalancer Service and rolled out new versions without downtime.
• Jenkins: tied all of it together, with credentials, parameters and upstream and downstream jobs.

That is the everyday work of a DevOps engineer, end to end. From here, good next steps are GitOps with Argo CD (the cluster pulls changes from Git instead of Jenkins pushing them), packaging manifests with Helm, and monitoring the running app with Prometheus.

To see how all the pieces fit together again, or to revisit any course, go back to the DevOps Bootcamp roadmap. And remember the habit from day one: delete the cluster when you stop practising.

Official references

• Jenkins: Pipeline Build Step (build job)
• Jenkins: Pipeline Syntax (parameters, when, post)
• Kubernetes: kubectl rollout status

Written By: Shramik Masti