How to Prevent Two Pipeline Jenkins Jobs of the Same Type to Run in Parallel on the Same Node?

1. Introduction

In Jenkins, we need to control how our pipeline jobs run. We often need to ensure that our system uses resources properly and that conflicts between jobs are avoided.

One common scenario we face is the need to prevent two pipeline jobs of the same type from running in parallel on the same node. When jobs run concurrently on one node, they might compete for CPU, memory, or disk I/O. They may also share the same workspace, leading to file conflicts or even inconsistent outcomes.

In this tutorial, we explain several methods we can use to control job concurrency on a single node.

2. Understanding the Need to Restrict Parallel Execution

Running multiple instances of the same pipeline on one node can lead to several problems.

First, resource contention is a major concern. When two jobs use a lot of memory or CPU simultaneously, they may slow each other down or even fail if the node runs out of resources. For example, suppose one job runs a heavy build process while another job on the same node performs resource-intensive tests. In that case, we might see significant delays or failures due to insufficient memory or CPU allocation.

Second, there is a risk of workspace conflicts. By default, Jenkins uses the same workspace directory for all job builds. This means that if two builds run concurrently, files might be overwritten or become corrupted. We have experienced situations where simultaneous writes led to unstable behavior in our build process, making it hard to track down the source of errors. These issues may result in unpredictable build outcomes and make debugging very challenging.

Third, modifying shared resources can cause data corruption. If our jobs change shared resources or update common files, one job might interfere with another. Such interference could lead to corrupted outputs, failing tests, or even security vulnerabilities if sensitive data is modified unexpectedly.

The overall goal is to ensure that only one instance of a particular type of job runs on a given node at any time while still allowing multiple jobs to run on different nodes concurrently. This approach helps us maintain a robust and efficient continuous integration pipeline.

3. Using the disableConcurrentBuilds Option

Jenkins offers a built-in way to prevent concurrent executions: using the disableConcurrentBuilds option in a declarative pipeline. This option guarantees that if one instance of our pipeline runs, no new instance will start until the current one finishes. This method is straightforward and works regardless of which node the job is running on.

Below is a code sample that we can copy and paste into our Jenkins file:

pipeline {
    agent any
    options {
        disableConcurrentBuilds()
    }
    stages {
        stage('Build') {
            steps {
                // Insert build steps here
            }
        }
        stage('Test') {
            steps {
                // Insert test steps here
            }
        }
    }
}

In the sample above, the disableConcurrentBuilds option stops a new pipeline instance from starting until the running instance is completed. This approach applies globally and does not differentiate between nodes. This means that even if we have multiple nodes available, only one build of the job can run at any time. While this method is simple and effective, it might be too strict for environments where we want to allow parallel builds on different nodes.

4. Utilizing the Throttle Concurrent Builds Plugin

For more granular control, especially when we want to allow parallel builds across nodes but not on a single node, the Throttle Concurrent Builds Plugin offers the necessary flexibility. This plugin will enable us to group jobs into categories and limit how many jobs from the same category can run concurrently on a node. By grouping jobs and setting the maximum number of concurrent builds per node to one, we ensure that only one job from a category is active on any single node.

4.1. Install the Plugin

The first step is to install the plugin:

1. Go to Manage Jenkins and then Manage Plugins
2. In the Available tab, search for the Throttle Concurrent Builds Plugin
3. Select the plugin and install it without restarting if possible

Installing the plugin is straightforward. Once installed, it provides additional configuration options not available by default in Jenkins.

4.2. Configure Throttle Categories

After installing the plugin, we need to configure a throttle category. This category will define how many builds can run on a node for jobs that belong to it:

1. Go to Manage Jenkins and then Configure System
2. Scroll down to the section labeled Throttle Concurrent Builds
3. Under Categories, click the button to add a new category
4. Enter a name for the category, for example, myThrottleCategory
5. Set the Maximum Concurrent Builds Per Node to 1. This setting ensures that no more than one job from this category runs on a single node
6. Optionally, we can set the Maximum Total Concurrent Builds to control the overall number of builds running across all nodes
7. Another option is to set the Maximum Concurrent Builds Per Node Labeled, which allows us to specify the name of the nodes that should have limitation

By setting these limits, we give ourselves precise control over job execution. This allows us to run multiple builds concurrently across different nodes while ensuring that no node is overloaded by running more than one instance of the same job.

The final configuration will look similar to this one:

Those of us using a declarative pipeline can also integrate these settings into our Jenkinsfile using the throttleJobProperty option:

pipeline {
    agent any
    options {
        throttleJobProperty(
            categories: ['MyThrottleCategory'],
            throttleEnabled: true,
            throttleOption: 'category'
        )
    }
    stages {
        stage('Build') {
            steps {
                // Insert build steps here
            }
        }
        stage('Test') {
            steps {
                // Insert test steps here
            }
        }
    }
}

In this configuration, the throttleJobProperty option links the job to MyThrottleCategory. The plugin enforces the rule that only one job from this category can run on any given node.

5. Implementing Node-Level Locks with the Lockable Resources Plugin

Another approach to controlling concurrent job execution is to use the Lockable Resources Plugin. This plugin gives us a mechanism to lock resources, such as a node or a specific resource on a node before a job runs. Doing so ensures that only one job can access the locked resource.

After installing the plugin, we need to define a resource the job will lock:

1. Go to Manage Jenkins and then Configure System
2. Scroll down to the section labeled Lockable Resources
3. Click the option to add a new lockable resource
4. Give the resource a unique name, for example, MyJobResource
5. Optionally add labels or a description to help us identify the resource
6. Save the configuration

Once the resource is defined, we modify our pipeline script to use the lock step. The lock step ensures that our job obtains an exclusive lock on the resource before executing a critical section of the code. Below is a code sample that we can copy and paste into our Jenkinsfile:

pipeline {
    agent any
    stages {
        stage('Preparation') {
            steps {
                lock(resource: 'MyJobResource') {
                    // This block is locked. Only one job on the same node can run this section at a time.
                    echo 'Lock acquired, running the critical section'
                    // Place critical operations here
                }
            }
        }
        stage('Other Steps') {
            steps {
                // Other steps that do not require locking can run in parallel.
                echo 'Running other operations'
            }
        }
    }
}

6. Best Practices

When choosing a method for controlling job concurrency, we should consider our overall Jenkins environment and the specific requirements of our jobs. For example, the disableConcurrentBuilds option is simple and effective when our needs are straightforward. Still, its global nature may be too strict for environments where we want to run parallel builds on different nodes.

The Throttle Concurrent Builds Plugin offers us more granular control by letting us create categories and set limits on concurrent builds per node. This flexibility is advantageous in larger environments where nodes have different capacities or other types of jobs require distinct handling.

Similarly, the Lockable Resources Plugin allows us to protect only the critical sections of our pipeline. This can be particularly beneficial when we need to prevent parallel execution for only a small part of the build process rather than the entire job. It also allows us to manage locks for shared resources that might be used by various jobs across our system.

Please remember that we need to balance the need for concurrency control with the desire for efficiency. Over-restricting job execution may lead to increased build times and unnecessary delays. We should strive to implement only the level of restriction needed to maintain stability and reliability.

7. Conclusion

In this article, we have discussed three methods to prevent two pipeline jobs of the same type from running in parallel on the same node.

We started with the disableConcurrentBuilds option, a simple built-in solution that applies globally. We then explored the Throttle Concurrent Builds Plugin, which allows us to allow parallel jobs on different nodes while limiting them to the same node. Finally, we looked at the Lockable Resources Plugin, which provides a fine-grained locking mechanism for critical sections of our pipeline.