Using a Jenkins Pipeline to Checkout Multiple Git Repos Into Same Job

1. Overview

Jenkins allows the automation of different stages of software delivery. It provides multiple integrations with common tools that we need during these stages. One of these integrations is working with Git repositories. We usually need to checkout and work with Git repositories in our Jenkins pipelines.

In this tutorial, we’ll learn how to use Jenkins to checkout Git repositories in our pipeline. We’ll also discuss how to checkout the repository into a subdirectory and how to checkout multiple repositories in the same pipeline job.

2. Checkout Repository Using git Step

The simplest way to checkout a repository in Jenkins is by using the git step.

The git step is part of the Jenkins Git plugin, which allows using fundamental Git operations inside Jenkins jobs.

The git step accepts the repository URL and branch name as arguments, and it checks out the code in the current directory:

pipeline{
    agent any
    stages{
        stage("checkout repo"){
            steps{
                git url: "https://github.com/javaee-samples/javaee7-samples" , branch: "master"
            }
        }
    }
}

Here, we provide the git step with the repository URL in the url parameter and the branch name master in the branch parameter.

Now, we’ll list the contents of the directory after we trigger our pipeline:

$ ls
Dockerfile  batch        ejb          jacc      jaxrpc  jca  jsf   pom.xml     validation
LICENSE     cdi          el           jaspic    jaxrs   jms  json  servlet     websocket
README.md   concurrency  interceptor  javamail  jaxws   jpa  jta   test-utils

As we can see, the repository contents under the master branch are now checked out to our directory.

3. Checkout Repository Into Subdirectory Using dir Step

Instead of executing the pipeline steps into the current directory, we can specify a subdirectory using a dir step.

The dir step accepts a string literal value, which becomes the name of a subdirectory that it creates. So, any pipeline steps written inside the dir step will be executed under this directory.

Let’s check this with an example:

pipeline{
    agent any
    stages{
        stage("create subdirectory"){
            steps{
                dir("myPipelineDirectory"){
                    sh "touch myfile.txt"
                }
            }
        }
    }
}

In the above code, we provide a string value of myPipelineDirectory to the dir step. So, it creates a new subdirectory with this name under the current directory. Also, inside the dir step, we touch a new file with the name myfile.txt.

Now, let’s see the contents of the current directory after we triggered the pipeline:

$ ls
myPipelineDirectory
$ cd myPipelineDirectory
$ ls
myfile.txt

As we can see, a subdirectory with the name myPipelineDirectory is created, and inside this subdirectory, we can see the new myfile.txt.

So now we can checkout our repository in a subdirectory by including our git step inside a dir step:

pipeline{
    agent any
    stages{
        stage("create subdirectory"){
            steps{
                dir("myCheckoutDirectory"){
                    git url: "https://github.com/javaee-samples/javaee7-samples" , branch: "master"
                }
            }
        }
    }
}

Now, let’s check our directory contents after triggering the pipeline:

$ ls
myCheckoutDirectory
$ cd myCheckoutDirectory/
$ ls
Dockerfile  cdi          interceptor  jaxrpc  jms   jta         validation
LICENSE     concurrency  jacc         jaxrs   jpa   pom.xml     websocket
README.md   ejb          jaspic       jaxws   jsf   servlet
batch       el           javamail     jca     json  test-utils

Again, we can see a new subdirectory is created, and our repository is checked out inside this subdirectory.

4. Checkout Repository Into Subdirectory Using checkout Step

Another way of checking out a Git repository into a subdirectory is by using a checkout step.

The checkout step is an alternative to the git step, but it provides more functionality that allows us to customize the checkout operation. Just as with most of the Jenkins steps, we can modify the functionality of the checkout step by passing arguments.

One of the arguments that we can pass to the checkout step is a relativeTargetDir argument. This argument specifies a subdirectory that we create to checkout the repository into.

Let’s check this with an example:

pipeline{
    agent any
    stages{
        stage("create subdirectory for checkout"){
            steps{
                 checkout([$class: 'GitSCM',
                         branches: [[name: "master"]],
                         doGenerateSubmoduleConfigurations: false,
                         extensions: [[$class: 'RelativeTargetDirectory',
                         relativeTargetDir: 'MyNewCheckoutDirectory']],
                         submoduleCfg: [],
                         userRemoteConfigs: [[url: "https://github.com/javaee-samples/javaee7-samples"]]
                         ])
                }
            }
        }
    }

In the above code, we can see that the checkout step accepts multiple arguments. The branches argument specifies which branch we need to checkout from the repository. Here, we provide the master branch. The relativeTargetDir argument specifies the name of a subdirectory in which we need to checkout the repository contents. Here, we name it MyNewCheckoutDirectory. Finally, the userRemoteConfigs argument specifies the URL of the repository.

So now let’s check our current directory after triggering the pipeline:

$ ls
MyNewCheckoutDirectory 
$ cd MyNewCheckoutDirectory/
$ ls
Dockerfile  batch        ejb          jacc      jaxrpc  jca  jsf   pom.xml     validation
LICENSE     cdi          el           jaspic    jaxrs   jms  json  servlet     websocket
README.md   concurrency  interceptor  javamail  jaxws   jpa  jta   test-utils

As we can see, the checkout step created the subdirectory and checked out the repository inside it.

5. Checkout Multiple Repositories Into Subdirectories

We can use the previous methods to checkout multiple Git repositories in a separate subdirectory for each one in the same pipeline job. To do that, we simply create a function that executes the code block for the checkout, and we pass the dynamic parameters like subdirectory name, repository URL, and branch to this function.

Let’s check an example using the checkout step:

def myCheckoutFunction(branch, subdirectory, URL){
        checkout([$class: 'GitSCM',
                         branches: [[name: branch]],
                         doGenerateSubmoduleConfigurations: false,
                         extensions: [[$class: 'RelativeTargetDirectory',
                         relativeTargetDir: subdirectory]],
                         submoduleCfg: [],
                         userRemoteConfigs: [[url: URL]]
                         ])
                    }
pipeline{
    agent any
    stages{
        stage("create subdirectory"){
            steps{
                script{
                        myCheckoutFunction("master", "myFirstRepo", "https://github.com/javaee-samples/javaee7-samples")
                        myCheckoutFunction("master", "mySecondRepo", "https://github.com/javaee-samples/javaee8-samples.git")
                        myCheckoutFunction("master", "myThirdRepo", "https://github.com/javaee-samples/microprofile1.2-samples.git")
                }
                }
            }
        }
    }

In the above code, we use the checkout step again to checkout our repository inside a subdirectory. However, we write the code for the checkout step inside a function that we define, which is the myCheckoutFunction(). We also specify the branch name, subdirectory, and URL as arguments for the function. Then, we use them inside the checkout step. Finally, we call the function in our pipeline multiple times for different repositories and subdirectories.

Let’s check our directory after triggering the pipeline:

$ ls
myFirstRepo  myThirdRepo mySecondRepo
$ cd myFirstRepo/
$ ls
Dockerfile  batch        ejb          jacc      jaxrpc  jca  jsf   pom.xml     validation
LICENSE     cdi          el           jaspic    jaxrs   jms  json  servlet     websocket
README.md   concurrency  interceptor  javamail  jaxws   jpa  jta   test-utils
$ cd ..
$ cd mySecondRepo/
$ ls
LICENSE    cdi  jaxrs  jsf     jsonb    security  test-utils
README.md  ejb  jpa    json-p  pom.xml  servlet   validation
$ cd ..
$ cd myThirdRepo/
$ ls
LICENSE  README.md  config  fault-tolerance  jwt-auth  metrics  pom.xml  test-utils

And we can see that all our subdirectories are created with the respective repository content inside each one.

We can also do the same using the dir step:

def myCheckoutFunction(branch, subdirectory, URL){
        dir(subdirectory){
            git branch: branch , url: URL
        }
                    }
pipeline{
    agent any
    stages{
        stage("create subdirectory"){
            steps{
                script{
                        myCheckoutFunction("master", "myFirstRepo", "https://github.com/javaee-samples/javaee7-samples")
                        myCheckoutFunction("master", "mySecondRepo", "https://github.com/javaee-samples/javaee8-samples.git")
                        myCheckoutFunction("master", "myThirdRepo", "https://github.com/javaee-samples/microprofile1.2-samples.git")
                }
                }
            }
        }
    }

Here, we use the same function but with the dir and git steps to checkout the repositories into the subdirectories.

Now, let’s check again after triggering our pipeline:

$ ls
myFirstRepo myThirdRepo mySecondRepo
$ cd myFirstRepo/
$ ls
Dockerfile  batch        ejb          jacc      jaxrpc  jca  jsf   pom.xml     validation
LICENSE     cdi          el           jaspic    jaxrs   jms  json  servlet     websocket
README.md   concurrency  interceptor  javamail  jaxws   jpa  jta   test-utils
$ cd ..
$ cd mySecondRepo/
$ ls
LICENSE    cdi  jaxrs  jsf     jsonb    security  test-utils
README.md  ejb  jpa    json-p  pom.xml  servlet   validation
$ cd ..
$ cd myThirdRepo/
$ ls
LICENSE  README.md  config  fault-tolerance  jwt-auth  metrics  pom.xml  test-utils

And we have the same result as before.

6. Conclusion

In this article, we learned how to checkout multiple Git repositories into separate subdirectories in the same Jenkins pipeline.

We saw how to use the git step as a simple way to checkout a repository. We used the dir step to create a subdirectory for our Jenkins pipeline execution.

Then, we used the git and dir steps together to checkout the repository in the subdirectory.

We also discussed how to use the checkout step to add more functionality to our checkout operation and create a subdirectory for the checkout.

Full Archive

About Baeldung