Mask Sensitive Data in Logs With Logback

1. Overview

In the new GDPR-present world, among many concerns, we must give special attention to logging individuals’ sensitive data. With the large amount of data being logged, it’s important to mask users’ sensitive details when logging.

In this tutorial, we’ll see how to mask sensitive data in logs with Logback. While this method serves as a final line of defense for our log files, it’s not considered the definitive solution to the problem.

2. Logback

Logback is one of the most widely used logging frameworks in the Java Community. It replaces Log4j, its predecessor. Logback offers a faster implementation, more configuration options, and more flexibility in archiving old log files.

Sensitive data is any information that is meant to be protected from unauthorized access. This can include anything from personally identifiable information (PII), such as Social Security numbers, to banking information, login credentials, address, email, and others.

We’ll mask the sensitive data that belongs to users while logging into our application logs.

3. Masking Data

Let’s say we log user details in the context of a web request. We need to mask the sensitive data related to users. Let’s assume our application receives the following request or response that we logged:

{
    "user_id": "87656",
    "ssn": "786445563",
    "address": "22 Street",
    "city": "Chicago",
    "Country": "U.S.",
    "ip_address":"192.168.1.1",
    "email_id":"[email protected]"
 }

Here, we can see that we have sensitive data like ssn, address, ip_address, and email_id. Hence, we have to mask this data while logging.

We’ll mask the logs centrally by configuring masking rules for all log entries produced by Logback. To do that, we have to implement a custom ch.qos.logback.classic.PatternLayout.

3.1. PatternLayout

The idea behind the configuration is to extend every Logback appender we need with a custom layout. In our case, we’ll write a MaskingPatternLayout class as an implementation of PatternLayout. Each mask pattern represents the regular expression that matches one type of sensitive data.

Let’s build the MaskingPatternLayout class:

public class MaskingPatternLayout extends PatternLayout {

    private Pattern multilinePattern;
    private List<String> maskPatterns = new ArrayList<>();

    public void addMaskPattern(String maskPattern) {
        maskPatterns.add(maskPattern);
        multilinePattern = Pattern.compile(maskPatterns.stream().collect(Collectors.joining("|")), Pattern.MULTILINE);
    }

    @Override
    public String doLayout(ILoggingEvent event) {
        return maskMessage(super.doLayout(event));
    }

    private String maskMessage(String message) {
        if (multilinePattern == null) {
            return message;
        }
        StringBuilder sb = new StringBuilder(message);
        Matcher matcher = multilinePattern.matcher(sb);
        while (matcher.find()) {
            IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
                if (matcher.group(group) != null) {
                    IntStream.range(matcher.start(group), matcher.end(group)).forEach(i -> sb.setCharAt(i, '*'));
                }
            });
        }
        return sb.toString();
    }
}

The implementation of PatternLayout.doLayout() is responsible for masking matched data in each log message of our application if it matches one of the configured patterns.

The maskPatterns list from logback.xml constructs a multiline pattern. Unfortunately, the Logback engine does not support constructor injection. If it comes as a list of properties, addMaskPattern is invoked for every config entry. So, we have to compile the pattern every time we add a new regex to the list.

3.2. Configuration

In general, we can use regex patterns to mask sensitive user details.

For example, for the SSN, we can use a regex like:

\"SSN\"\s*:\s*\"(.*)\"

And for the address, we can use:

\"address\"\s*:\s*\"(.*?)\" 

Furthermore, for the IP address data pattern (192.169.0.1), we can use the regex:

(\d+\.\d+\.\d+\.\d+)

Finally, for email, we can write:

([\w.-]+@[\w.-]+\.\w+)

Now, we’ll add these regex patterns in maskPattern tags inside our logback.xml file:

<configuration>
    <appender name="mask" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
           <layout class="com.baeldung.logback.MaskingPatternLayout">
	       <maskPattern>\"SSN\"\s*:\s*\"(.*?)\"</maskPattern> <!-- SSN JSON pattern -->
	       <maskPattern>\"address\"\s*:\s*\"(.*?)\"</maskPattern> <!-- Address JSON pattern -->
	       <maskPattern>(\d+\.\d+\.\d+\.\d+)</maskPattern> <!-- Ip address IPv4 pattern -->
	       <maskPattern>([\w.-]+@[\w.-]+\.\w+)</maskPattern> <!-- Email pattern -->
	       <pattern>%-5p [%d{ISO8601,UTC}] [%thread] %c: %m%n%rootException</pattern>
            </layout>
        </encoder>
    </appender>
</ configuration>

3.3. Execution

Now, we’ll create the JSON for the above example and use logger.info() to log the details:

Map<String, String> user = new HashMap<String, String>();
user.put("user_id", "87656");
user.put("SSN", "786445563");
user.put("address", "22 Street");
user.put("city", "Chicago");
user.put("Country", "U.S.");
user.put("ip_address", "192.168.1.1");
user.put("email_id", "[email protected]");
JSONObject userDetails = new JSONObject(user);

logger.info("User JSON: {}", userDetails);

After executing this, we can see the output:

INFO  [2021-06-01 16:04:12,059] [main] com.baeldung.logback.MaskingPatternLayoutExample: User JSON: 
{"email_id":"*******************","address":"*********","user_id":"87656","city":"Chicago","Country":"U.S.", "ip_address":"***********","SSN":"*********"}

Here, we can see that the user JSON in our logger has been masked:

{
    "user_id":"87656",
    "ssn":"*********",
    "address":"*********",
    "city":"Chicago",
    "Country":"U.S.",
    "ip_address":"*********",
    "email_id":"*****************"
 }

With this approach, we can only mask those data in log files for which we’ve defined regular expressions in maskPattern in logback.xml.

4. Conclusion

In this tutorial, we covered how to use the PatternLayout feature to mask sensitive data in application logs with Logback and how to add regex patterns in logback.xml to mask specific data.