Check if Spark Dataframe Is Empty

1. Introduction

When working with Apache Spark, we often encounter situations where a DataFrame becomes empty after applying transformations, filters, or joins.

Before triggering expensive actions or saving data to external storage, it’s a best practice to verify if the DataFrame contains any rows. While this may seem like a trivial check, different approaches trigger different Spark actions and can have a noticeable impact on performance, especially for large datasets.

In this tutorial, we’ll explore different ways to check whether a Spark DataFrame is empty.

2. Setup

First, let’s create the following sample DataFrame:

public static Dataset getDataFrame(SparkSession spark) {
    List<Row> players = List.of(
      RowFactory.create(1, "Messi", "Argentina"),
      RowFactory.create(2, "Ronaldo", "Portugal"),
      RowFactory.create(3, "Mbappe", "France"));
    return spark.createDataFrame(players, PLAYER_SCHEMA);
}

We’ll use this DataFrame throughout the examples.

3. Using the isEmpty() method

Since Spark 2.4.0, the easiest way to check whether a Dataset is empty is by using the isEmpty() method.

Let’s create an empty DataFrame by filtering for a country that doesn’t exist in the dataset.

Dataset<Row> englandPlayers = allPlayers.filter(col("country").equalTo("England"));
Assertions.assertTrue(englandPlayers.isEmpty());

Since there are no England players in the dataframe, the filter returns an empty dataframe and isEmpty() returns true.

4. Using the count() method

Another way to check for emptiness is by counting the number of rows in a DataFrame and checking if it’s zero:

Dataset<Row> englandPlayers = allPlayers.filter(col("country").equalTo("England"));
Assertions.assertEquals(0, englandPlayers.count());

Here, we invoked count() and verified that the number of rows is 0.

5. Using the takeAsList() Method

Another approach is to retrieve at most one row using takeAsList():

Dataset<Row> englandPlayers = allPlayers.filter(col("country").equalTo("England"));
Assertions.assertEquals(0, englandPlayers.takeAsList(1).size());

Here we called the takeAsList(1), which returns at most one row. We can then verify that the returned list is empty by checking whether its size is 0. Since only one row is requested, Spark can stop processing once it finds the first matching row, making this approach more efficient than counting every row.

Alternatively, we can also check if the list is empty instead of size():

Assertions.assertTrue(englandPlayers.takeAsList(1).isEmpty());

6. Comparing Methods for Performance

Although all of these approaches can determine whether a DataFrame is empty, they don’t have the same performance characteristics.The isEmpty() method is the recommended approach because it’s specifically designed for this purpose. Internally, Spark only needs to determine whether at least one row exists, allowing it to stop processing as soon as a row is found.

Similarly, takeAsList(1) requests only a single row. Once Spark retrieves the first row, it can terminate the scan, making this approach much more efficient than counting every row. In contrast, count() computes the total number of rows in the DataFrame. Since Spark must process the entire dataset to produce an exact count, this approach is considerably more expensive for large DataFrames.

In general, we should prefer isEmpty() whenever it’s available. If we’re working with an older Spark version that doesn’t support it, takeAsList(1).isEmpty() is a good alternative. We should reserve count() for situations where we also need the total number of rows.

7. Conclusion

In this article, we explored several ways to determine whether a Spark DataFrame is empty. We started with the isEmpty() method, which is the simplest and most expressive solution, and then looked at alternatives based on count() and takeAsList().

In most situations, isEmpty() is the preferred choice because it’s both expressive and efficient. For older Spark versions, takeAsList(1).isEmpty() provides a practical alternative, while count() is best reserved for cases where the total number of rows is also needed.

As always, the sample code used in this article is available over on GitHub.