FIFO Queue Support in Spring Cloud AWS

1. Overview

FIFO (First-In-First-Out) queues in AWS SQS are designed to ensure that messages are processed in the exact order they are sent and that each message is delivered only once.

Spring Cloud AWS v3 supports this functionality with easy-to-use abstractions that allow developers to handle FIFO queue features like message ordering and deduplication with minimal boilerplate code.

In this tutorial, we’ll explore three practical use cases of FIFO queues in the context of a financial transaction processing system:

• Ensuring strict message ordering for transactions within the same account
• Processing transactions from different accounts in parallel while maintaining FIFO semantics for each account
• Handling message retries in case of processing failures, ensuring that retries respect the original message order

We’ll demonstrate these scenarios by setting up an event-driven application and creating live tests to assert the behavior is as expected, leveraging the environment and test setup from the Spring Cloud AWS SQS V3 introductory article.

2. Dependencies

To begin, we’ll manage dependencies and ensure version compatibility by importing the Spring Cloud AWS Bill of Materials (BOM):

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws</artifactId>
            <version>3.2.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Next, we add the necessary Spring Cloud AWS starters for core functionality and SQS integration:

<dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-aws-starter</artifactId>
</dependency>
<dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-aws-starter-sqs</artifactId>
</dependency>

We’ll also include the Spring Boot Web Starter. Since we’re using the Spring Cloud AWS BOM, we don’t need to specify its version:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Finally, for testing, we’ll add dependencies for LocalStack and TestContainers with JUnit 5, Awaitility for asynchronous operation verification, and the Spring Boot Test Starter:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-testcontainers</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>localstack</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.awaitility</groupId>
    <artifactId>awaitility</artifactId>
    <scope>test</scope>
</dependency>

3. Setting Up the Local Test Environment

Next, we’ll configure our local test environment using Testcontainers and LocalStack. We’ll create a SqsLiveTestConfiguration class:

@Configuration
public class SqsLiveTestConfiguration {
    private static final String LOCAL_STACK_VERSION = "localstack/localstack:3.4.0";

    @Bean
    @ServiceConnection
    LocalStackContainer localStackContainer() {
        return new LocalStackContainer(DockerImageName.parse(LOCAL_STACK_VERSION));
    }
}

In this class, we declare our LocalStack test container as a Spring Bean and use the @ServiceConnection annotation to handle the wiring on our behalf.

4. Setting Up the Queue Names

We’ll define our SQS queue names within the application.yml file, leveraging Spring Boot’s configuration externalization features:

events:
  queues:
    fifo:
      transactions-queue: "transactions-queue.fifo"
      slow-queue: "slow-queue.fifo"
      failure-queue: "failure-queue.fifo"

This structure organizes our queue names under a hierarchical structure, making it easy to manage and access them in our application code. The .fifo suffix is mandatory for FIFO queues in SQS.

5. Setting Up the Application

Let’s illustrate these concepts with a practical example using a Transaction microservice. This service will process TransactionEvent messages, representing financial transactions that must be kept in order within each account.

First, we define our Transaction entity:

public record Transaction(UUID transactionId, UUID accountId, double amount, TransactionType type) {}

Along with a TransactionType enum:

public enum TransactionType {
    DEPOSIT,
    WITHDRAW
}

Next, we create the TransactionEvent:

public record TransactionEvent(UUID transactionId, UUID accountId, double amount, TransactionType type) {
    public Transaction toEntity() {
        return new Transaction(transactionId, accountId, amount, type);
    }
}

The TransactionService class handles the processing logic and maintains a simulated repository for testing purposes:

@Service
public class TransactionService {
    private static final Logger logger = LoggerFactory.getLogger(TransactionService.class);

    private final ConcurrentHashMap<UUID, List<Transaction>> processedTransactions = 
      new ConcurrentHashMap<>();

    public void processTransaction(Transaction transaction) {
        logger.info("Processing transaction: {} for account {}",
          transaction.transactionId(), transaction.accountId());
        processedTransactions.computeIfAbsent(transaction.accountId(), k -> new ArrayList<>())
          .add(transaction);
    }

    public List<Transaction> getProcessedTransactionsByAccount(UUID accountId) {
        return processedTransactions.getOrDefault(accountId, new ArrayList<>());
    }
}

6. Processing Events in Order

In our first scenario, we’ll create a listener that processes events and will create a test to assert that we receive the events in the same order as they were sent. We’ll use the @RepeatedTest annotation to run the test 100 times to make sure it’s consistent, and see how it behaves with a Standard SQS queue instead of a FIFO.

6.1. Creating the Listener

Let’s create our first listener to receive and process events in order.  We’ll use the @SqsListener annotation, leveraging Spring’s placeholder resolution to resolve the queue name from the application.yml file:

@Component
public class TransactionListener {
    private final TransactionService transactionService;

    public TransactionListener(TransactionService transactionService) {
        this.transactionService = transactionService;
    }

    @SqsListener("${events.queues.fifo.transactions-queue}")
    public void processTransaction(TransactionEvent transactionEvent) {
        transactionService.processTransaction(transactionEvent.toEntity());
    }
}

Notice that no further setup is necessary. Behind the scenes the framework will detect that queue type is FIFO and make all necessary adjustments to make sure the listener method receives the messages in the correct order.

6.2. Creating the Test

Let’s create a test that asserts that the order in which the messages are received is exactly the order in which were sent. We start with a test suite that extends the BaseSqsLiveTest we created earlier:

@SpringBootTest
public class SpringCloudAwsSQSTransactionProcessingTest extends BaseSqsLiveTest {
    @Autowired
    private SqsTemplate sqsTemplate;

    @Autowired
    private TransactionService transactionService;

    @Value("${events.queues.fifo.transactions-queue}")
    String transactionsQueue;

    @Test
    void givenTransactionsFromSameAccount_whenSend_shouldReceiveInOrder() {
        var accountId = UUID.randomUUID();
        var transactions = List.of(createDeposit(accountId, 100.0), 
          createWithdraw(accountId, 50.0), createDeposit(accountId, 25.0));
        var messages = createTransactionMessages(accountId, transactions);

        sqsTemplate.sendMany(transactionsQueue, messages);

        await().atMost(Duration.ofSeconds(5))
          .until(() -> transactionService.getProcessedTransactionsByAccount(accountId),
            isEqual(eventsToEntities(transactions)));
    }
}

In this test, we’re leveraging SqsTemplate‘s sendMany() method that enables us to send up to 10 messages in the same batch. We’re then waiting for up to 5 seconds to receive the messages in order.

We’ll also create a few helper methods to help us keep the test logic clean. The sendMany() method expects a List<Pojo>, so the createTransactionMessages() method maps each transaction for an accountId to a message:

private List<Message<TransactionEvent>> createTransactionMessages(UUID accountId,
        Collection<TransactionEvent> transactions) {
    return transactions.stream()
        .map(transaction -> MessageBuilder.withPayload(transaction)
          .setHeader(SqsHeaders.MessageSystemAttributes.SQS_MESSAGE_GROUP_ID_HEADER,
             accountId.toString())
          .build())
        .toList();
}

In SQS FIFO, the MessageGroupId attribute is used to inform which messages should be grouped together and received in order.  In our scenario, we must make sure that the transactions for one Account are kept in order, but we don’t need any ordering between accounts, so we’ll use the accountId as the MessageGroupId. For that we can use the headers in SqsHeaders and the framework will map them to SQS Message Attributes:

The remaining helper methods are straightforward methods for mapping events to transactions, and creating TransactionEvents:

private List<Transaction> eventsToEntities(List<TransactionEvent> transactionEvents) {
    return transactionEvents.stream()
      .map(TransactionEvent::toEntity)
      .toList();
}

private TransactionEvent createWithdraw(UUID accountId1, double amount) {
    return new TransactionEvent(UUID.randomUUID(), accountId1, amount, TransactionType.WITHDRAW);
}

private TransactionEvent createDeposit(UUID accountId1, double amount) {
    return new TransactionEvent(UUID.randomUUID(), accountId1, amount, TransactionType.DEPOSIT);
}

6.3. Running the Test

When we run the tests, we’ll see that the test passes and generates logs similar to this, with the transactions happening in the same order as we declared them:

TransactionService : Processing transaction: DEPOSIT:100.0 for account f97876f9-5ef9-4b62-a69d-a5d87b5b8e7e
TransactionService : Processing transaction: WITHDRAW:50.0 for account f97876f9-5ef9-4b62-a69d-a5d87b5b8e7e
TransactionService : Processing transaction: DEPOSIT:25.0 for account f97876f9-5ef9-4b62-a69d-a5d87b5b8e7e

If we’re still not convinced and want to make sure it’s not a coincidence, we can add the @RepeatableTest annotation to run the tests 100 times

@RepeatedTest(100)
void givenTransactionsFromSameAccount_whenSend_shouldReceiveInOrder() {
   // ...test remains the same
}

All 100 runs should equally pass with the logs in the same order.

For an extra sanity check, let’s use a standard queue rather than a FIFO and verify how it behaves.

For that, we’ll need to remove the .fifo suffix from the queue name in application.yml:

transactions-queue: "transactions-queue"

Next, we’ll comment out the code that adds the MessageId header in the createTransactionMessages() method since Standard SQS Queues do not support that attribute:

// .setHeader(SqsHeaders.MessageSystemAttributes.SQS_MESSAGE_GROUP_ID_HEADER, accountId.toString())

Now, let’s run the test 100 times again. You’ll notice that the test will sometimes pass because coincidentally the messages arrived in the expected order, but other times it’ll fail since there’s no guarantee about message ordering in a Standard Queue.

Before we wrap up this section, let’s undo these changes and add the .fifo suffix in the queue, remove the @RepeatedTest annotation, and uncomment the MessageGroupId code.

7. Processing Multiple Message Groups in Parallel

In SQS FIFO, in order to maximize message consumption throughput, we can process messages from different message groups in parallel, while keeping message order within a message group. Spring Cloud AWS SQS supports that behavior out of the box, with no further configuration needed.

To illustrate that behavior, let’s add a method to TransactionService that simulates a slow connection:

public void simulateSlowProcessing(Transaction transaction) {
    try {
        processTransaction(transaction);
        Thread.sleep(Thread.sleep(100));
        logger.info("Transaction processing completed: {}:{} for account {}",
          transaction.type(), transaction.amount(), transaction.accountId());
    } catch (InterruptedException e) {
        Thread.currentThread()
          .interrupt();
        throw new RuntimeException(e);
    }
}

The slow connection will help us assert that the messages from different accounts are being processed in parallel while preserving transaction order within each account.

Now, let’s create a listener that’ll use the new method in the TransactionListener class:

@SqsListener("${events.queues.fifo.slow-queue}")
public void processParallelTransaction(TransactionEvent transactionEvent) {
    transactionService.simulateSlowProcessing(transactionEvent.toEntity());
}

Lastly, let’s create a test to assert the behavior:

@Test
void givenTransactionsFromDifferentAccounts_whenSend_shouldProcessInParallel() {
    var accountId1 = UUID.randomUUID();
    var accountId2 = UUID.randomUUID();

    var account1Transactions = List.of(createDeposit(accountId1, 100.0),
      createWithdraw(accountId1, 50.0), createDeposit(accountId1, 25.0));
    var account2Transactions = List.of(createDeposit(accountId2, 50.0),
      createWithdraw(accountId2, 25.0), createDeposit(accountId2, 50.0));

    var allMessages = Stream.concat(createTransactionMessages(accountId1, account1Transactions).stream(),
      createTransactionMessages(accountId2, account2Transactions).stream()).toList();

    sqsTemplate.sendMany(slowQueue, allMessages);

    await().atMost(Duration.ofSeconds(5))
      .until(() -> transactionService.getProcessedTransactionsByAccount(accountId1),
        isEqual(eventsToEntities(account1Transactions)));

    await().atMost(Duration.ofSeconds(5))
      .until(() -> transactionService.getProcessedTransactionsByAccount(accountId2),
        isEqual(eventsToEntities(account2Transactions)));
}

In this test, we’re sending two sets of transaction events for two different accounts. We’re again leveraging the sendMany() method to send all messages in the same batch, and asserting the messages are being received in the expected order.

When we run the test, we should see logs similar to this:

TransactionService : Processing transaction: DEPOSIT:50.0 for account 639eba64-a40d-458a-be74-2457dff9d6d1
TransactionService : Processing transaction: DEPOSIT:100.0 for account 1a813756-520c-4713-a0ed-791b66e4551c
TransactionService : Transaction processing completed: DEPOSIT:100.0 for account 1a813756-520c-4713-a0ed-791b66e4551c
TransactionService : Transaction processing completed: DEPOSIT:50.0 for account 639eba64-a40d-458a-be74-2457dff9d6d1
TransactionService : Processing transaction: WITHDRAW:50.0 for account 1a813756-520c-4713-a0ed-791b66e4551c
TransactionService : Processing transaction: WITHDRAW:25.0 for account 639eba64-a40d-458a-be74-2457dff9d6d1
TransactionService : Transaction processing completed: WITHDRAW:50.0 for account 1a813756-520c-4713-a0ed-791b66e4551c
TransactionService : Transaction processing completed: WITHDRAW:25.0 for account 639eba64-a40d-458a-be74-2457dff9d6d1
TransactionService : Processing transaction: DEPOSIT:50.0 for account 639eba64-a40d-458a-be74-2457dff9d6d1
TransactionService : Processing transaction: DEPOSIT:25.0 for account 1a813756-520c-4713-a0ed-791b66e4551c

We can see that both accounts are being processed in parallel while keeping order within each account, which is also verified by the test passing.

8. Retrying Processing in Order

In our last scenario, we’ll simulate a network failure and ensure the processing order remains consistent. When the listener method throws an error, the framework halts execution for that message group and does not acknowledge the messages. SQS serves the remaining messages again after the visibility window has expired.

To illustrate that behavior, we’ll add a new method to our TransactionService that’ll always fail the first time it processes a message.

First, let’s add a Set to hold the IDs that have already failed:

private final Set<UUID> failedTransactions = ConcurrentHashMap.newKeySet();

Then let’s add the processTransactionWithFailure() method:

public void processTransactionWithFailure(Transaction transaction) {
    if (!failedTransactions.contains(transaction.transactionId())) {
        failedTransactions.add(transaction.transactionId());
        throw new RuntimeException("Simulated failure for transaction " +
          transaction.type() + ":" + transaction.amount());
    }
    processTransaction(transaction);
}

This method will throw an error the first time a transaction is processed but will process it normally in subsequent retries.

Now, let’s add the listener to process the messages. We’ll set messageVisibilitySeconds to one to diminish the visibility window and speed up retries in our test:

@SqsListener(value = "${events.queues.fifo.failure-queue}", messageVisibilitySeconds = "1")
public void retryFailedTransaction(TransactionEvent transactionEvent) {
    transactionService.processTransactionWithFailure(transactionEvent.toEntity());
}

Lastly, let’s create a test to assert the behavior is as expected:

@Test
void givenTransactionProcessingFailure_whenSend_shouldRetryInOrder() {
    var accountId = UUID.randomUUID();
    var transactions = List.of(createDeposit(accountId, 100.0),
      createWithdraw(accountId, 50.0), createDeposit(accountId, 25.0));
    var messages = createTransactionMessages(accountId, transactions);

    sqsTemplate.sendMany(failureQueue, messages);

    await().atMost(Duration.ofSeconds(10))
      .until(() -> transactionService.getProcessedTransactionsByAccount(accountId),
        isEqual(eventsToEntities(transactions)));
}

In this test, we’re sending three events and asserting they’re processed in the expected order.

When we run the test, we should see a log similar to this in the exception stack trace:

Caused by: java.lang.RuntimeException: Simulated failure for transaction DEPOSIT:100.0

Followed by:

TransactionService : Processing transaction: DEPOSIT:100.0 for account 3f684ccb-80e8-4e40-9136-c3b59bdd980b

Indicating the event was successfully processed on the second try.

We should see 2 similar pairs for the next events:

Caused by: java.lang.RuntimeException: Simulated failure for transaction WITHDRAW:50.0
TransactionService : Processing transaction: WITHDRAW:50.0 for account 3f684ccb-80e8-4e40-9136-c3b59bdd980b
Caused by: java.lang.RuntimeException: Simulated failure for transaction DEPOSIT:25.0
TransactionService : Processing transaction: DEPOSIT:25.0 for account 3f684ccb-80e8-4e40-9136-c3b59bdd980b

This indicates that the events were processed in the right order, even in the presence of failures.

9. Conclusion

In this article, we explored Spring Cloud AWS v3’s support for FIFO queues. We created a transaction processing service that relies on the events being processed in order and asserted message order is respected in three different scenarios: processing a single message group, processing multiple message groups in parallel, and retrying messages after a failure.

We tested each scenario by setting up a local test environment and creating live tests to assert our logic.