Making Integration Testing Easier With TestContainers-scala

Last updated: March 18, 2024

Reviewed by: Dominique Plante

1. Introduction

Testcontainers-scala is a Scala library that enables us to instantiate Docker containers when running functional and integration tests. It’s a Scala wrapper around Testcontainers-java, with out-of-the-box integrations with ScalaTest and MUnit.

In this tutorial, we’ll see how Testcontainers-scala works and how we can integrate it into our ScalaTest-based test classes.

2. What Are Integration and Functional Tests?

When testing complex applications, we often find ourselves wanting to mock various “infrastructural” dependencies. Some examples might be relational databases, services such as ElasticSearch, Redis, application servers, or AWS services, such as S3 buckets.

Furthermore, we often like to test our application with the services we’ll use in the real world. In these cases, we can leverage Testcontainers-scala.

Generally speaking, integration tests validate the behavior of groups of individual software modules, rather than testing such modules in isolation (as unit tests do).

Functional tests, on the other hand, are a type of black-box testing validating the requirements of an application.

In other words, integration tests describe how different components should interact with one another, whereas functional tests describe what the system does as a whole.

3. Testcontainers-scala

Testcontainers-scala lets us write self-contained integration tests that make use of instantiating Docker containers to mock dependencies.

In particular, our tests can use any resource that’s deployed as a Docker image. Furthermore, we can build our own Docker images and use them in our tests. This is useful, for example, when our system comprises several applications that we want to test together.

3.1. Dependencies and Setup

In this article, we’re going to use Testcontainers-scala with ScalaTest.

First, we need to import the testcontainers-scala-scalatest module:

libraryDependencies += "com.dimafeng" %% "testcontainers-scala-scalatest" % "0.40.15" % IntegrationTest

In sbt, we store integration tests in src/it/scala/.  sbt doesn’t support integration tests by default, so we have to do that explicitly:

lazy val scala_libraries_4 = (project in file("scala-libraries-4"))
  .configs(IntegrationTest)
  .settings(
    Defaults.itSettings,
    IntegrationTest / fork := true)

We need the last line in the snippet above to enable sbt forking so that sbt runs the tests in a separate JVM. Using the code above, the containers shut down gracefully when the tests terminate.

Lastly, we need Docker installed in order to run containers.

3.2. Main Classes and Traits

Testcontainers-scala provides us with four traits that we can mix in into our test classes:

  • TestContainerForAll starts a single container before all the test cases in the class and stops it after all tests terminate. This roughly corresponds to starting up the container in a beforeAll() method and shutting it down in an afterAll() method.
  • TestContainersForAll is similar to TestContainerForAll, but it starts up more than one container.
  • TestContainerForEach starts a single container before each test case in the class and stops it after each test. This roughly corresponds to starting up the container in a beforeEach() method and shutting it down in an afterEach() method.
  • TestContainersForEach is similar to TestContainerForEach, but it starts up more than one container.

The choice among those traits strongly depends on what we’re testing. Starting up a fresh container for each test case provides much more isolation guarantees, at the cost of increased test duration.

Testcontainers-scala models Docker containers using two classes:

  • ContainerDef is a blueprint for a Docker container and defines how to build the latter. A ContainerDef is somewhat similar to a Dockerfile description, as it defines how the container should be run. ContainerDef exposes a start() method, starting up the container and returning an instance of Container.
  • Container represents a started Docker container and exposes several methods for getting some properties of the container itself, such as the exposed ports.

4. Defining the Code to Be Tested

We’ll see how to use Testcontainers-scala to test code that handles uploading files to an AWS S3 bucket. In the examples in this tutorial, we’ll make use of LocalStack to avoid using a real AWS account.

Let’s start by importing the AWS SDK for S3 into our project:

libraryDependencies += "software.amazon.awssdk" % "s3" % "2.20.71"

Then, let’s define a simple uploader class:

class SimpleS3Uploader(
  region: String,
  endpoint: URI,
  accessKeyId: String,
  secretAccessKey: String
) {
  private lazy val s3 = S3Client
    .builder()
    .region(Region.of(region))
    .endpointOverride(endpoint)
    .credentialsProvider(
      StaticCredentialsProvider.create(
        AwsBasicCredentials.create(accessKeyId, secretAccessKey)
      )
    )
    .forcePathStyle(true)
    .build()

  def upload(bucket: String, filePath: Path): Unit = {
    s3.putObject(
      PutObjectRequest
        .builder()
        .bucket(bucket)
        .key(filePath.getFileName.toString)
        .build(),
      filePath
    )
  }
}

SimpleS3Uploader simply instantiates an S3 client and uses it to upload a file to a bucket, using the S3Client::putObject() method.

When building the client, we specify the target AWS region and override the target endpoint to use LocalStack’s one. We then pass the credentials and force the client to use path-style addressing for buckets, which is needed by LocalStack.

The upload() method simply specifies the name of the bucket and the name of the uploaded file in the bucket (via the key() method of PutObjectRequest‘s builder). Lastly, we input a Path to the file to upload.

5. Test Definitions

In this section, we’re going to see different ways to integrate Testcontainers-scala into our tests.

Using Testcontainers-scala, we can create containers using one of the following three approaches:

  • using pre-defined test modules: useful when we want to mock the dependency on widely used services, such as databases or web servers
  • programmatically writing the definitions of our containers: useful when there are no pre-defined test modules for the services we need or when we want to customize a service
  • using a Docker Compose file: useful when we have lots of services to mock

In our test suites, we can either commit to defining the containers only in one of the three approaches above or mix them up.

For example, we could define some containers programmatically (to mock dependencies on other services we developed, for instance), but keep using pre-defined modules whenever possible.  We should be aware that the more things we mix together, the harder it will be to figure out how they all work together later.

5.1. Using Pre-defined Modules

Most of the time we can use one of the predefined modules to simplify the integration of Testcontainers-scala in our tests. First, let’s import the LocalStack module:

libraryDependencies ++= Seq(
  "com.amazonaws" % "aws-java-sdk-s3" % "1.12.474" % IntegrationTest,
  "com.dimafeng" %% "testcontainers-scala-localstack-v2" % "0.40.15" % IntegrationTest
)

Since the LocalStack module still uses Version 1 of the AWS S3 SDK, we also had to import it.

We can now define our test class, mixing in one of the four traits we saw earlier. In our examples, we’ll use TestContainerForEach.

The first thing we have to do is to define our test container:

override val containerDef: LocalStackV2Container.Def =
  LocalStackV2Container.Def(
    tag = "1.3.0",
    services = Seq(Service.S3)
  )

The definition states the version of the LocalStack Docker image (in this case, 1.3.0), and the services we want to use. The test module takes care of exposing the container port for us and makes sure to start the tests once the container is ready.

We can use the withContainers() method, defined by Testcontainers-scala, to access our LocalStack container in the test definitions:

"SimpleS3Uploader" should "upload a file in the desired bucket" in {
  withContainers { ls =>
    new SimpleS3Uploader(
      region = ls.region.toString,
      endpoint = ls.endpointOverride(Service.S3),
      accessKeyId = ls.container.getAccessKey,
      secretAccessKey = ls.container.getSecretKey
    ).upload(
      BucketName,
      Paths.get(getClass.getClassLoader.getResource("s3-test.txt").toURI)
    )
  }
}

In the example above, ls has type LocalStackV2Container and provides us with useful runtime information about the LocalStack container, such as its endpoint and the working AWS region.

LocalStackV2Container is simply a wrapper around a Java class, JavaLocalStackContainer, exposing much more information. We can access it through the container property, as we do to retrieve the AWS secret access key and access key ID.

5.2. Using Custom Containers

When we can’t use a pre-defined test module, Testcontainers-scala lets us define our own containers. We might also want to do that to have more flexibility on the generated container. Even though it’s not a good testing practice, we might want, for example, to have full control of the port exposed by the LocalStack container.

We should keep in mind that Testcontainers-scala allocates random ports to the containers to avoid conflicts. If we allocate the ports ourselves and map multiple containers to the same port, or map a container to a port that’s already in use, the test suite gets aborted.

To use our own container definitions we have to extend the GenericContainer class:

class MyLocalStackContainer private (
  hostPort: Int,
  underlying: GenericContainer
) extends GenericContainer(underlying) {
  underlying.container.setPortBindings(
    List(s"$hostPort:$LocalStackPort").asJava
  )

  val endpoint = s"http://localhost:$hostPort"
  val accessKeyId = "not_used"
  val secretAccessKey = "not_used"
}

MyLocalStackContainer inputs a port number and a GenericContainer, used to invoke the constructor of the GenericContainer class. The latter acts a bit as a copy constructor, extracting the Java container from underlying and using it to instantiate a new GenericContainer.

Then, MyLocalStackContainer invokes the setPortBindings() method to bind the hostPort parameter to LocalStackPort (more on this below). This allows us to control exactly which port on the host will be mapped to the one exposed by the LocalStack container.

Lastly, MyLocalStackContainer defines a few convenience values, MyLocalStackContainer::endpoint, MyLocalStackContainer::accessKeyId, and MyLocalStackContainer::secretAccessKey. MyLocalStackContainer::endpoint is particularly interesting, as it explicitly declares the endpoint used to make calls to the LocalStack container.

Let’s see how to create an instance of GenericContainer:

object MyLocalStackContainer {
  private val LocalStackPort = 4566

  case class Def(hostPort: Int)
    extends GenericContainer.Def[MyLocalStackContainer](
      new MyLocalStackContainer(
        hostPort,
        GenericContainer(
          dockerImage = "localstack/localstack:1.3.0",
          exposedPorts = Seq(LocalStackPort),
          waitStrategy = new LogMessageWaitStrategy().withRegEx(".*Ready\\.\n")
        )
      )
    )
}

The MyLocalStackContainer object defines a case class, named Def in compliance with Testcontainers-scala’s naming convention, extending GenericContainer.Def.

The case class initializes an instance of MyLocalStackContainer, specifying some information about the actual container, such as the Docker image, the exposed port, and a strategy to detect that the container is ready. This is necessary because Testcontainers-scala has to know when the target container has completed the start-up in order to trigger the execution of the tests. In this case, we’re determining this by inspecting the logs, looking for the string “Ready”.

Lastly, let’s see what the test body looks like:

"SimpleS3Uploader" should "upload a file in the desired bucket" in {
  withContainers { ls =>
    val region = "us-east-1"

    new SimpleS3Uploader(
      region = region,
      endpoint = new URI(ls.endpoint),
      accessKeyId = ls.accessKeyId,
      secretAccessKey = ls.secretAccessKey
    ).upload(
      BucketName, 
      Paths.get(getClass.getClassLoader.getResource("s3-test.txt").toURI)
    )
  }
}

As before, we’re using withContainers() to fetch information about the running LocalStack container. The difference now is that we can only invoke the properties we’ve defined in MyLocalStackContainer, i.e., endpoint, accessKeyId, and secretAccessKey.

This approach is surely more complex than the one using pre-defined modules but, as we saw, gives us much more flexibility.

5.3. Using a Docker Compose File

The third way to integrate Testcontainers-scala in our repository is via a Docker Compose file. Let’s first create this file:

version: "3.8"
services:
  localstack:
    image: localstack/localstack:1.3.0
    restart: always
    ports:
      - "5000:4566"

The Docker Compose file defines a new container for LocalStack. It maps port 4566 of the container to 5000 on the host.

Next, let’s use DockerComposeContainer in our Scala tests to start up the containers defined via Docker Compose:

private val ExposedPort = 5000

override val containerDef: DockerComposeContainer.Def =
  DockerComposeContainer.Def(
    new File(this.getClass.getClassLoader.getResource("docker-compose.yml").getFile),
    exposedServices = Seq(
      ExposedService("localstack", ExposedPort, new LogMessageWaitStrategy().withRegEx(".*Ready\\.\n"))
    )
  )

In the example above, we specified the path to the Docker Compose file and described the exposed services. To do so, we have to state the name of the service, the port on the host (which must match the one mapped in the Docker Compose file), and how to detect that the container is ready, as we did before.

Let’s see how to write a test:

"SimpleS3Uploader" should "upload a file in the desired bucket" in {
  val region = "us-east-1"
  val endpoint = s"http://localhost:$ExposedPort"
  val file = new File(
    getClass.getClassLoader.getResource("s3-test.txt").getFile
  )
  new SimpleS3Uploader(
    region = region,
    endpoint = new URI(endpoint),
    accessKeyId = "not_used",
    secretAccessKey = "not_used"
  ).upload(
    BucketName, 
    file.toPath
  )
}

There’s no need to use withContainers() anymore, as Testcontainers-scala cannot provide us with the same information as before. Hence, we have to hardcode the exposed port, the region, and the credentials by hand.

This approach might come in handy when we have many containers to mock and don’t need much information about the way they’re deployed (such as ports, endpoints, and so on). It might also be useful as a first step to integrate Testcontainers-scala in our code base.

6. Conclusion

In this article, we dove into Testcontainers-scala, a Scala wrapper around a Java library aiming at simplifying the way we mock external services in our integration tests.

First, we saw how to set up the library. Secondly, we saw which classes and traits are available for us to use in our integration tests. Lastly, we analyzed three different ways to integrate Testcontainers-scala in our project, each one with different pros and cons.

Using Docker containers in our tests, instead of depending on third-party hosted services, has several advantages. First, we don’t have to pay fees to use external services. Secondly, tests run faster, as everything runs on the same machine and don’t have to go through a network.

Generally speaking, we could use Docker containers without the need for Testcontainers-scala. The latter library, however, dramatically simplifies the definition of the containers and the programmatic interaction with them.

As usual, the code for the examples is available over on GitHub.

Comments are open for 30 days after publishing a post. For any issues past this date, use the Contact form on the site.