## 1. Introduction

The Maxout activation function was proposed by Ian Goodfellow in the paper “Maxout Networks” in 2013. It involves employing multiple ReLU functions over the input and selecting the maximum value as the output. Mathematically, it is represented as:

(1)

where is the input, , and are the weights and biases of the ReLU activation functions.

## 2. Custom TensorFlow Implementation of Maxout

In TensorFlow, we can define a custom maxout implementation:

```
import tensorflow as tf
import numpy as np
def maxout(inputs, num_units, axis=None):
shape = inputs.shape.as_list()
if axis is None:
axis = -1
num_channels = shape[axis]
if num_channels % num_units:
raise ValueError('number of features({}) is not a multiple of num_units({})'
.format(num_channels, num_units))
shape[axis] = -1
shape += [num_channels // num_units]
outputs = tf.reduce_max(tf.reshape(inputs, shape), axis=-1, keepdims=False)
return outputs
```

**The function takes three main arguments: inputs, num_units, and axis**. The

*inputs*parameter represents the input tensor on which the maxout operation is applied. The

*num_units*parameter specifies the number of units or neurons in the maxout layer. The

*axis*parameter indicates the axis along which the max pooling operation is performed. By default, if the

*axis*is not specified, the function assumes that the last dimension corresponds to the channels or features and performs the max pooling operation along this axis.

After reshaping, we apply the *tf.reduce_max* operation along the specified axis. This operation calculates the maximum value along the given axis, effectively performing the maxout operation.

## 3. Comparison with the B**uilt-in Library of Maxout**

Here, we use TensorFlow’s *tf.reshape* function to reshape the input tensor into a shape where the number of elements in each group is inferred. Then, we apply max pooling within each group to mimic the maxout operation:

```
# Call custom maxout function
output_tensor_custom = maxout(input_tensor, num_units=2, axis=-1)
# Reshape input tensor to facilitate maxout operation
input_tensor_reshaped = tf.reshape(input_tensor, (2, 3, -1))
# Apply tf.reduce_max to mimic Maxout
output_tensor_builtin = tf.reduce_max(tf.reshape(input_tensor_reshaped, (2, 3, -1, 3)), axis=-1)
```

Then, we use *tf.reduce_max* to find the maximum value along the last dimension of the reshaped tensor, effectively performing max pooling within each group. The *axis=-1* argument indicates that the max pooling is performed along the last dimension of the tensor.

## 4. Total Code Comparison

Below is the entire code sample:

```
import tensorflow as tf
import numpy as np
def maxout(inputs, num_units, axis=None):
shape = inputs.shape.as_list()
if axis is None:
axis = -1
num_channels = shape[axis]
if num_channels % num_units:
raise ValueError('number of features({}) is not a multiple of num_units({})'
.format(num_channels, num_units))
shape[axis] = -1
shape += [num_channels // num_units]
outputs = tf.reduce_max(tf.reshape(inputs, shape), axis=-1, keepdims=False)
return outputs
input_tensor = tf.constant(np.random.randn(2, 3, 6)) # Example input tensor of shape (2, 3, 6)
output_tensor_custom = maxout(input_tensor, num_units=2, axis=-1)
input_tensor_reshaped = tf.reshape(input_tensor, (2, 3, -1))
output_tensor_builtin = tf.reduce_max(tf.reshape(input_tensor_reshaped, (2, 3, -1, 3)), axis=-1)
print("Output tensor from custom maxout:")
print(output_tensor_custom.numpy())
print("\nOutput tensor mimicking Maxout using tf.reduce_max:")
print(output_tensor_builtin.numpy())
if np.allclose(output_tensor_custom.numpy(), output_tensor_builtin.numpy()):
print("\nCustom maxout function and tf.reduce_max produce the same output.")
else:
print("\nCustom maxout function and tf.reduce_max produce different output.")
```

## 5. Conclusion

In this article, we discussed the maxout activation that offers several advantages such as improved efficiency and robustness, it also comes with computational costs and challenges in hyperparameter tuning.

We implemented and utilized the maxout activation function in the Python-based machine learning framework TensorFlow.