What Is Cost Function in Neural Network?

As a data scientist or a software engineer working with neural networks, you might have come across the term “cost function” or “loss function” quite often. A cost function is a mathematical function that measures how well a neural network is performing on a specific task. In this article, we will discuss the concept of cost function in neural networks and its importance.

Table of Contents

1. Importance of Cost Functions
2. Types of Cost Functions
3. Choosing the Right Cost Function
4. Conclusion

Why Cost Function is Important

The main goal of any neural network is to make accurate predictions. A cost function helps to quantify how far the neural network’s predictions are from the actual values. It is a measure of the error between the predicted output and the actual output. The cost function plays a crucial role in training a neural network. During the training process, the neural network adjusts its weights and biases to minimize the cost function. The goal is to find the minimum value of the cost function, which corresponds to the best set of weights and biases that make accurate predictions.

Types of Cost Functions

There are different types of cost functions, and the choice of cost function depends on the type of problem being solved. Here are some commonly used cost functions:

Mean Squared Error (MSE)

The mean squared error is one of the most popular cost functions for regression problems. It measures the average squared difference between the predicted and actual values. The formula for MSE is:

MSE = (1/n) * Σ(y - ŷ)^2

Where:

• n is the number of samples in the dataset
• y is the actual value
• ŷ is the predicted value

Python Code Example:

from sklearn.metrics import mean_squared_error

actual_values = [2, 4, 5, 7]
predicted_values = [1.5, 3.5, 4.5, 7.5]

mse = mean_squared_error(actual_values, predicted_values)
print("Mean Squared Error:", mse)

Output:

Mean Squared Error: 0.25

Binary Cross-Entropy

The binary cross-entropy cost function is used for binary classification problems. It measures the difference between the predicted and actual values in terms of probabilities. The formula for binary cross-entropy is:

Binary Cross-Entropy = - (1/n) * Σ(y * log(ŷ) + (1 - y) * log(1 - ŷ))

Where:

• n is the number of samples in the dataset
• y is the actual value (0 or 1)
• ŷ is the predicted probability (between 0 and 1)

Python Code Example:

import numpy as np

def binary_cross_entropy(y_true, y_pred):
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

actual_values = np.array([1, 0, 1, 0])
predicted_values = np.array([0.9, 0.1, 0.8, 0.2])

bce = binary_cross_entropy(actual_values, predicted_values)
print("Binary Cross-Entropy:", bce)

Output:

Binary Cross-Entropy: 0.164252033486018

Categorical Cross-Entropy

The categorical cross-entropy cost function is used for multi-class classification problems. It measures the difference between the predicted and actual values in terms of probabilities. The formula for categorical cross-entropy is:

Categorical Cross-Entropy = - (1/n) * ΣΣ(y(i,j) * log(ŷ(i,j)))

Where:

• n is the number of samples in the dataset
• y(i,j) is the actual value of the i-th sample for the j-th class
• ŷ(i,j) is the predicted probability of the i-th sample for the j-th class

Python Code Example:

from keras.losses import categorical_crossentropy
import numpy as np

actual_values = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
predicted_values = np.array([[0.9, 0.1, 0.0], [0.0, 0.8, 0.2], [0.1, 0.2, 0.7]])

cce = categorical_crossentropy(actual_values, predicted_values)
print("Categorical Cross-Entropy:", cce.numpy().mean())

Output:

Categorical Cross-Entropy: 0.22839300363692283

There are some other loss functions such as

Hinge Loss, which is often employed in support vector machines and binary classification tasks within neural networks, emphasizing correct classification by penalizing misclassifications based on their proximity to the decision boundary. It is particularly advantageous for linear classifiers and exhibits robustness against outliers.

Another noteworthy loss function is Sparse Categorical Cross-Entropy, serving as a memory-efficient alternative for multi-class classification without the need for one-hot encoding of target variables. On the other hand, Kullback-Leibler Divergence (KL Divergence) finds application as a regularization term in variational autoencoders, promoting a closer match between the learned and target distributions to enhance generalization and mitigate overfitting.

Lastly, the Cramér Loss has emerged as a valuable tool for addressing domain adaptation challenges, leveraging the Cramér distance to encourage alignment between source and target distributions. This is particularly beneficial in scenarios involving different data distributions, such as transfer learning applications.

How to Choose a Cost Function

Choosing the right cost function is crucial for the performance of a neural network. Here are some factors to consider when choosing a cost function:

Type of Problem

The type of problem being solved determines the type of cost function to use. For example, regression problems require a different cost function than classification problems.

Output Activation Function

The output activation function can also influence the choice of cost function. For example, if the output activation function is sigmoid, then the binary cross-entropy cost function is a good choice. If the output activation function is softmax, then the categorical cross-entropy cost function is a good choice.

Network Architecture

The network architecture can also influence the choice of cost function. For example, if the network has multiple outputs, then the multi-task loss function is a good choice.

Conclusion

In conclusion, a cost function is a crucial component of a neural network. It measures the error between the predicted and actual values and helps the network to adjust its weights and biases to make accurate predictions. There are different types of cost functions, and the choice of cost function depends on the type of problem being solved. When choosing a cost function, consider factors such as the type of problem, output activation function, and network architecture. By choosing the right cost function, you can improve the performance of your neural network and make accurate predictions.

SHARE:

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Request a demo today to learn more.