Expand Authors Top

If you have a few years of experience in the Java ecosystem and you’d like to share that with the community, have a look at our Contribution Guidelines.

November Discount Launch 2022 – Top
We’re finally running a Black Friday launch. All Courses are 30% off until next Friday:

>> GET ACCESS NOW

Expanded Audience – Frontegg – Security (partner)
announcement - icon User management is very complex, when implemented properly. No surprise here.

Not having to roll all of that out manually, but instead integrating a mature, fully-fledged solution - yeah, that makes a lot of sense.
That's basically what Frontegg is - User Management for your application. It's focused on making your app scalable, secure and enjoyable for your users.
From signup to authentication, it supports simple scenarios all the way to complex and custom application logic.

Have a look:

>> Elegant User Management, Tailor-made for B2B SaaS

NPI – Lightrun – Spring (partner)

We rely on other people’s code in our own work. Every day. It might be the language you’re writing in, the framework you’re building on, or some esoteric piece of software that does one thing so well you never found the need to implement it yourself.

The problem is, of course, when things fall apart in production - debugging the implementation of a 3rd party library you have no intimate knowledge of is, to say the least, tricky. It’s difficult to understand what talks to what and, specifically, which part of the underlying library is at fault.

Lightrun is a new kind of debugger.

It's one geared specifically towards real-life production environments. Using Lightrun, you can drill down into running applications, including 3rd party dependencies, with real-time logs, snapshots, and metrics. No hotfixes, redeployments, or restarts required.

Learn more in this quick, 5-minute Lightrun tutorial:

>> The Essential List of Spring Boot Annotations and Their Use Cases

1. Introduction

By default, any errors encountered during a Spring Batch job processing will make a corresponding step fail. However, there are many situations where we'd rather like to skip the currently processed item for certain exceptions.

In this tutorial, we'll explore two approaches to configure skip logic in the Spring Batch framework.

2. Our Use Case

For the purpose of examples, we'll reuse a simple, chunk-oriented job presented already in our Spring Batch introductory article.

This job converts some financial data from a CSV to XML format.

2.1. Input Data

First, let's add a few rows to the original CSV file:

username, user_id, transaction_date, transaction_amount
devendra, 1234, 31/10/2015, 10000
john, 2134, 3/12/2015, 12321
robin, 2134, 2/02/2015, 23411
, 2536, 3/10/2019, 100
mike, 9876, 5/11/2018, -500
, 3425, 10/10/2017, 9999

As we can see, the last three rows contain some invalid data – rows 5 and 7 are missing the username field, and the transaction amount in row 6 is negative.

In the later sections, we'll configure our batch job to skip these corrupted records.

3. Configuring Skip Limit and Skippable Exceptions

3.1. Using skip and skipLimit

Let's now discuss the first of two ways to configure our job to skip items in case of a failure — the skip and skipLimit methods:

@Bean
public Step skippingStep(
  ItemProcessor<Transaction, Transaction> processor,
  ItemWriter<Transaction> writer) throws ParseException {
    return stepBuilderFactory
      .get("skippingStep")
      .<Transaction, Transaction>chunk(10)
      .reader(itemReader(invalidInputCsv))
      .processor(processor)
      .writer(writer)
      .faultTolerant()
      .skipLimit(2)
      .skip(MissingUsernameException.class)
      .skip(NegativeAmountException.class)
      .build();
}

First of all, to enable skip functionality, we need to include a call to faultTolerant() during the step-building process.

Within skip() and skipLimit(), we define the exceptions we want to skip and the maximum number of skipped items.

In the above example, if either a MissingUsernameException or NegativeAmountException is thrown during read, process, or write phase, then the currently processed item will be omitted and counted against the total limit of two.

Consequently, if any exception is thrown a third time, then the whole step will fail.

3.1. Using noSkip

In the previous example, any other exception besides MissingUsernameException and NegativeAmountException makes our step fail.

In some situations, however, it may be more appropriate to identify exceptions that should make our step fail and skip on any other.

Let's see how we can configure this using skip, skipLimit, and noSkip:

@Bean
public Step skippingStep(
  ItemProcessor<Transaction, Transaction> processor,
  ItemWriter<Transaction> writer) throws ParseException {
    return stepBuilderFactory
      .get("skippingStep")
      .<Transaction, Transaction>chunk(10)
      .reader(itemReader(invalidInputCsv))
      .processor(processor)
      .writer(writer)
      .faultTolerant()
      .skipLimit(2)
      .skip(Exception.class)
      .noSkip(SAXException.class)
      .build();
}

With the above configuration, we instruct Spring Batch framework to skip on any Exception (within a configured limit) except SAXException. This means SAXException always causes a step failure.

The order of the skip() and noSkip() calls doesn't matter.

4. Using Custom SkipPolicy

Sometimes we may need a more sophisticated skip-checking mechanism. For that purpose, Spring Batch framework provides the SkipPolicy interface.

We can then provide our own implementation of skip logic and plug it into our step definition.

Keeping the preceding example in mind, imagine we still want to define a skip limit of two items and make only MissingUsernameException and NegativeAmountException skippable.

However, an additional constraint is that we can skip NegativeAmountException, but only if the amount doesn't exceed a defined limit.

Let's implement our custom SkipPolicy:

public class CustomSkipPolicy implements SkipPolicy {

    private static final int MAX_SKIP_COUNT = 2;
    private static final int INVALID_TX_AMOUNT_LIMIT = -1000;

    @Override
    public boolean shouldSkip(Throwable throwable, int skipCount) 
      throws SkipLimitExceededException {

        if (throwable instanceof MissingUsernameException && skipCount < MAX_SKIP_COUNT) {
            return true;
        }

        if (throwable instanceof NegativeAmountException && skipCount < MAX_SKIP_COUNT ) {
            NegativeAmountException ex = (NegativeAmountException) throwable;
            if(ex.getAmount() < INVALID_TX_AMOUNT_LIMIT) {
                return false;
            } else {
                return true;
            }
        }

        return false;
    }
}

Now, we can use our custom policy in a step definition:

    @Bean
    public Step skippingStep(
      ItemProcessor<Transaction, Transaction> processor,
      ItemWriter<Transaction> writer) throws ParseException {
        return stepBuilderFactory
          .get("skippingStep")
          .<Transaction, Transaction>chunk(10)
          .reader(itemReader(invalidInputCsv))
          .processor(processor)
          .writer(writer)
          .faultTolerant()
          .skipPolicy(new CustomSkipPolicy())
          .build();
    }

And, similarly to our previous example, we still need to use faultTolerant() to enable skip functionality.

This time, however, we do not call skip() or noSkip(). Instead, we use the skipPolicy() method to provide our own implementation of the SkipPolicy interface.

As we can see, this approach gives us more flexibility, so it can be a good choice in certain use cases.

5. Conclusion

In this tutorial, we presented two ways to make a Spring Batch job fault-tolerant.

Even though using a skipLimit() together with skip() and noSkip() methods seems to be more popular, we may find implementing a custom SkipPolicy to be more convenient in some situations.

As usual, all the code examples are available over on GitHub.

November Discount Launch 2022 – Bottom
We’re finally running a Black Friday launch. All Courses are 30% off until next Friday:

>> GET ACCESS NOW

Generic footer banner
Comments are closed on this article!