How to Stop a Zombie Job on Jenkins Without Restarting the Server?

1. Overview

In Jenkins, a zombie job is a build stuck in an endless loop, consuming resources and potentially causing problems for other builds. It is important to stop zombie jobs immediately to prevent them from causing further problems on the Jenkins server.

In this tutorial, we’ll explain how to identify and stop a zombie job on Jenkins without restarting the server.

2. Understanding the Problem

A zombie job consists of an ongoing process that never ends. It can occur for various reasons, such as a build step that is stuck or a job unable to finish due to external dependencies. Zombie jobs can cause problems through resource consumption and the failure of other builds. Additionally, this makes it difficult to see the status of other builds. Thus, it is crucial to stop zombie jobs immediately to prevent such issues.

3. Creating a Zombie Job

In Jenkins, we can use infinite pipeline jobs to create zombie jobs by producing an error in the pipeline. To illustrate, let’s create a pipeline job for the same:

pipeline {
    agent any
    stages {
        stage('Infinite Loop') {
            steps {
                script {
                    while (true) {
                        println 'This is an infinite loop!'
                        Thread.sleep(10000)
                    }
                }
            }
        }
    }
}

In this pipeline job, a script step contains an infinite loop with a Thread.sleep(10000) call inside. The Thread.sleep() method will cause the script to pause for 10 seconds before continuing the loop. This will make it more difficult for the stop button to interrupt the build. On running this job, we’ll get the following error:

Scripts not permitted to use staticMethod java.lang.Thread sleep long. Administrators can decide whether to approve or reject this signature.

The admin user has the option of approving or rejecting this signature. The Jenkins server is configured to block the Thread.sleep() method in pipeline scripts. This security feature prevents malicious scripts from causing problems on the server. In order to approve the request, the admin user needs to follow these steps:

• Go to the Jenkins dashboard
• Click on Manage Jenkins in the Jenkins dashboard
• Click on In-process Script Approval to approve the Thread.sleep() signature

On building the job again, we won’t be able to stop this zombie job using the stop button.

4. Stopping a Zombie Job

In Jenkins, we can terminate a normal job using the stop button in the build view. Furthermore, this will terminate that build and release all its dependencies. But, if a job is stuck and can’t be stopped, it must be terminated forcefully.

4.1. Using the Finish Method

To stop a zombie job, Jenkins API can be used directly with the build. By using Jenkins builds finish method, we can mark a build as completed and assign it a result status. To run the script, we need to follow these steps:

• Go to the Jenkins dashboard
• Click on Manage Jenkins in the Jenkins dashboard
• Add the script below in Script Console

Jenkins.instance.getItemByFullName("sampleZombieJob")
  .getBuildByNumber(17)
  .finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build")); 

Run the following script into the Script Console to forcefully terminate the build.

4.2. Using the Thread Interrupt Method

We can also stop a zombie job using the Thread.interrupt() method from the Script Console:

Thread.getAllStackTraces().keySet().each() {
    if (it.name.contains('sampleZombieJob')) {  
        println "Stopping $it.name"
        it.interrupt()
    }
}

This script will interrupt the last build of sampleZombieJob and release all the resources it was consuming.

4.3. Using the Thread Stop Method

The Script Console also allows us to stop a zombie job using the Thread.stop() method:

Thread.getAllStackTraces().keySet().each() {
    if (it.name.contains('sampleZombieJob')) {  
        println "Stopping $it.name"
        it.stop()
    }
}

The script above will terminate the zombie job and release all its resources. The Thread.stop() approach may result in data loss and leave the server unstable. In general, we should stop the build by clicking the stop button on the build page or by using the interrupt() method. The Thread.stop() method should only be used as a final resort.

5. Conclusion

In this article, we learned how to stop a zombie job on Jenkins. First, we created a zombie job using the Jenkins pipeline. After that, we discussed possible solutions to kill the zombie process.

We can also find scripts to create and stop a zombie job over on GitHub.