1. Introduction
Machine learning models are widely classified into two types: parametric and nonparametric models.
In this tutorial, we’ll present these two types, analyze their different approaches, and examine the main models of each group as well as their benefits and drawbacks.
2. Parametric Models
By following certain assumptions, the workload of the training process for the model is simplified and lightened. Therefore by doing so, the model is able to learn certain patterns of the input without the need to train with large datasets. Additionally, the model’s efficiency and performance are determined by these hyperparameters that regulate its behavior.
2.1. Examples of Parametric Models
Some examples of parametric models in neural networks include linear or polynomial regression, which are straightforward models that imply that the input and output have a linear or polynomial relation respectively.
Also, logistic regression’s basic assumptions include the independence of the observations and lack of multicollinearity, and strongly influential outliers. Other parametric models are Gaussian mixtures and Hidden Markov models that suppose that the input data follows a Gaussian distribution or a Markov process with unobserved (hidden) states, respectively. Moreover, a feedforward neural network with one or more hidden layers belongs to parametric models.
2.2. Advantages and Disadvantages
On the other hand, often, the assumption simplifies the problem, and thus the model s unable to comprehend complex patterns and relations between the data. Furthermore, parametric models have proven to be sensitive to outliers, show limited performance to nonlinearity problems, and struggle to adapt to new unseen data.
3. Non-parametric Models
The second category includes non-parametric models. These models don’t need to make assumptions about the relations between the input and output to generate an outcome and also don’t require a certain number of parameters to be set and learned. Studies have shown that non-parametric perform better on large datasets and are more flexible.
3.1. Examples of Non-parametric Models
There are several models in this category as well.
Common non-parametric algorithms are the random forests or decision trees that split the input into a smaller space based on the data features, generating the prediction based on the class. Moreover, Support Vector Machines with non-linear kernels are non-parametric models that find a hyperplane and create a feature space that map the input and output data. Another example is the k-Nearest Neighbors (k-NN) algorithm that generates the outcomes based on the majority class of the k closest data for a single instance. Lastly, neural networks with non-parametric activation functions, for example, use a kernel activation or radial basis function activation. are considered non-parametric models.
3.2. Advantages and Disadvantages
On the other hand, non-parametric models require more data input in order to generate better predictions and therefore are computationally expensive as they need to estimate more parameters than parametric models. Also, they are usually harder to understand and analyze as there are no functional assumptions about the behavior of input and output.
4. Main Differences
The main differences between parametric and non-parametric models include the assumptions about the relationship between data and the fixed or not number of parameters about the data. Moreover, another difference is the data requirement each category demands and the computational complexity. Lastly, an additional distinction is the adaptability they show to new data. Note that these are general principles and that there may be exceptions based on the individual situation and dataset dealing with.
The main differences can be summarized in the table:
5. Conclusion
In this tutorial, we walked through parametric and non-parametric models. In particular, we introduced the two categories, talked about the most common examples, and analyzed their benefits and limitations.