1. Introduction
In this tutorial, we’ll review non-trainable parameters as they are commonly referred to in neural networks. We’ll look at what they are and how they function and then consider the benefits and drawbacks of using non-trainable parameters.
2. What Are the Parameters in a Neural Network?
A typical neural network usually comprises connections between neurons, weights and biases. These components function together to process data and produce some output.
Now, when talking about the parameters in a neural network, we usually refer to parameters learned during the training phase. These parameters are adjustable to the data and the task at hand and are often updated during training. The most common is the model weights.
While training a neural network, these weights are usually updated through backpropagation. Backpropagation is a technique to compute the gradients in a neural network. It is used to propagate errors back through the network in order to minimize the error.
3. Non-trainable Parameters
As already mentioned, parameters usually refer to the weights in a neural network. These parameters often come in two flavors, trainable and non-trainable parameters. Trainable parameters are those that have their values calculated and updated during the training phase. Whereas non-trainable parameters are parameters whose values are not updated during training.
Suppose we have a CNN that classifies RGB images as either cat or dog. The architecture consists of an input layer and convolution and pooling layers connected to a simple feed-forward network like an MLP. The main task of the convolution layers, i.e. layers prior to the feed-forward network, is to learn the features of cats and dogs from the input data.
After the features are learned, the MLP has the task of classifying images as cat or dog. Now during training, after a couple of iterations, we freeze the convolution layers or the portion that learns the features to focus on fine-tuning the MLP for classification. In this case, the convolutional layers are non-trainable and hence do not learn further. This is a typical example of the use of non-trainable parameters:
4. When Do We Use Them?
Non-trainable parameters are most commonly used in pre-trained models and transfer learning. In pre-training, a neural network is first trained as a model on one task or dataset. Then, the parameters or models from this training are used to train another model on a different task or dataset.
Some well-known examples of pre-trained models are BERT, VGG-16, ResNet50, Inceptionv3, and GPT-4.
In transfer learning, the model parameters are frozen and saved in the pre-trained model; hence, they are not trainable when applied to a new task. In this case, the model is only fine-tuned to the new task and new data. Hence, training does not begin from scratch.
5. Should We Use Non-trainable Parameters?
Whether or not to use non-trainable parameters depends on the model used and the task at hand. For example, a simple classification model will most likely not require the use of non-trainable parameters. Whereas a computer vision task for image segmentation might require the use of non-trainable parameters.
5.1. Pros
There are several benefits and drawbacks to using non-trainable parameters. We’ll start off with the benefits. Firstly, using non-trainable parameters shortens the time required to train models. Using non-trainable parameters means fewer parameter updates, hence faster training time.
5.2. Cons
The most notable drawback to using non-trainable parameters is that model parameters cannot be adjusted when they are set to non-trainable. In this case, no further learning takes place in the model.
6. Conclusion
In this article, we’ve reviewed non-trainable parameters. Non-trainable parameters are model parameters that are not updated during training. Most commonly, it refers to the weights in a neural network.
Using non-trainable parameters usually depends on the task for which the model is being used. Some models and tasks may require the use of non-trainable parameters, while others may not.
Non-trainable parameters have the benefit of optimizing training and making the process faster. However, this means that no further learning takes place in the model.