1. Overview
In this tutorial, we’ll talk about the term ablation study in the field of machine learning. First, we’ll briefly discuss the initial definition of the term in biology, and then we’ll focus on its meaning in the context of machine learning. Then, we’ll illustrate its importance and present two simple examples of an ablation study.
2. Ablation in Biology
The original meaning of ablation refers to the surgical removal of organs, abnormal growths, or harmful tissues from the body.
In our case, we will focus more on the term ablative brain surgery, which refers to the ablation of some brain tissues to treat a neurological disorder. It was first introduced in the field of experimental neuropsychology, where scientists removed some parts of the animals’ brains and studied the effects that this had on their behavior.
For example, if we remove a certain tissue of the animal’s brain and the animal cannot move its leg, we can assume that this region of the brain controls the movements of the leg.
3. Ablation Study in Machine Learning
Most times in machine learning, we implement models that consist of multiple different parts that influence the total performance. Therefore, it is very important to provide ways to measure the contribution of these parts to the overall model. This is where the concept of an ablation study comes where certain parts of the network are removed in order to gain a better understanding of the network’s behavior.
We can easily perceive the analogy between the ablation study and the ablative brain surgery that was mentioned previously. In the former, we remove some parts of the machine learning model, while in the latter, we remove some parts of the brain. In both cases, our goal is to measure causality.
In the image below, we can see an example of an ablation study in a model with N modules. Each time we remove one of the modules and check the performance of the new model to investigate the influence of the removed module:
4. Importance
Now, let’s discuss the importance of the ablation study in machine learning.
In a machine learning system, there are usually many different learning modules. Understanding the influence of each module is crucial, and ablation study is a low-effort way to investigate the causation of these parts.
For example, it costs nothing to just remove some layers of a neural network and run the same experiment without these layers to investigate their contribution to the model performance.
5. Examples
Finally, we’ll present two simple examples of ablation studies to better illustrate the concept.
5.1. Object Detection
In object detection, our goal is to locate and identify the depicted objects in an image. It is one of the most important tasks in computer vision, with applications ranging from autonomous driving and security to retail and healthcare.
A classic object detection system consists of three main modules:
- An object localization module that searches for the location of the objects in the image
- A feature extractor that is usually a large convolutional neural network and is applied in every detected object
- A classification module that classifies each object based on the computed feature
In an ablation study, we can train the whole model many times where we keep the feature extractor and the classification module the same and change only the architecture of the object localization module. In this way, we’ll investigate the influence of different object localization modules on the overall performance regardless of the rest modules of the system. In general, we can run experiments where we keep two of the three modules the same and only change the architecture of the third module.
5.2. LSTM ARCHITECTURE
LSTM is a special type of recurrent neural network where each neuron consists of three kinds of gates: forget, update, and output. A common question that arises in this architecture is how essential is each type of gate.
An ablation study here is to delete one of the gates and keep the others the same. There has been a great deal of work done in this direction, with the GRU architecture being one example.
6. Conclusion
In this tutorial, we presented the term ablation study. First, we introduced the term in the field of biology, and then we defined the term in the field of machine learning. Finally, we discussed the importance of ablation studies in machine learning along with two examples.