What is a loss function in simple words?
Asked Answered
M

3

20

Can anyone please explain in simple words and possibly with some examples what is a loss function in the field of machine learning/neural networks?

This came out while I was following a Tensorflow tutorial: https://www.tensorflow.org/get_started/get_started

Maelstrom answered 18/3, 2017 at 18:4 Comment(6)
I would rather suggest you to start reading a (basic) Machine Learning book instead of "How to use tensorflow". Tensorflow can't do magic if you don't know what you are doing.Cherub
Do you suggest any online material I can follow?Maelstrom
Probably the best machine learning course to follow. Or google other books from Andrew Ng, such us this one.Cherub
Or if you want more deep learning oriented maybe this one from Standford.Cherub
If you want a quick not too deep introduction, have a look at this youtube talk (2:30 hours) youtube.com/watch?v=vq2nnJ4g6N0Cherub
Yaser Abu-Mostafa's Caltech online course also has a pretty nice intro to learning theory: work.caltech.edu/telecourse.htmlKimkimball
S
16

The loss function is how you're penalizing your output.

The following example is for a supervised setting i.e. when you know the correct result should be. Although loss functions can be applied even in unsupervised settings.

Suppose you have a model that always predicts 1. Just the scalar value 1.

You can have many loss functions applied to this model. L2 is the euclidean distance.

If I pass in some value say 2 and I want my model to learn the x**2 function then the result should be 4 (because 2*2 = 4). If we apply the L2 loss then its computed as ||4 - 1||^2 = 9.

We can also make up our own loss function. We can say the loss function is always 10. So no matter what our model outputs the loss will be constant.

Why do we care about loss functions? Well they determine how poorly the model did and in the context of backpropagation and neural networks. They also determine the gradients from the final layer to be propagated so the model can learn.

As other comments have suggested I think you should start with basic material. Here's a good link to start off with http://neuralnetworksanddeeplearning.com/

Sogdian answered 18/3, 2017 at 18:12 Comment(0)
P
29

It describes how far off the result your network produced is from the expected result - it indicates the magnitude of error your model made on its prediciton.

You can then take that error and 'backpropagate' it through your model, adjusting its weights and making it get closer to the truth the next time around.

Placencia answered 18/3, 2017 at 18:15 Comment(1)
This is the clear, simple and concise definition that I have been looking for! Many articles actually miss these points, which are fundamental to someone approaching ML for the first timeEngedi
S
16

The loss function is how you're penalizing your output.

The following example is for a supervised setting i.e. when you know the correct result should be. Although loss functions can be applied even in unsupervised settings.

Suppose you have a model that always predicts 1. Just the scalar value 1.

You can have many loss functions applied to this model. L2 is the euclidean distance.

If I pass in some value say 2 and I want my model to learn the x**2 function then the result should be 4 (because 2*2 = 4). If we apply the L2 loss then its computed as ||4 - 1||^2 = 9.

We can also make up our own loss function. We can say the loss function is always 10. So no matter what our model outputs the loss will be constant.

Why do we care about loss functions? Well they determine how poorly the model did and in the context of backpropagation and neural networks. They also determine the gradients from the final layer to be propagated so the model can learn.

As other comments have suggested I think you should start with basic material. Here's a good link to start off with http://neuralnetworksanddeeplearning.com/

Sogdian answered 18/3, 2017 at 18:12 Comment(0)
S
3

Worth to note we can speak of different kind of loss functions: Regression loss functions and classification loss functions.

Regression loss function describes the difference between the values that a model is predicting and the actual values of the labels.

So the loss function has a meaning on a labeled data when we compare the prediction to the label at a single point of time.

This loss function is often called the error function or the error formula.

Typical error functions we use for regression models are L1 and L2, Huber loss, Quantile loss, log cosh loss.

Note: L1 loss is also know as Mean Absolute Error. L2 Loss is also know as Mean Square Error or Quadratic loss.

Loss functions for classification represent the price paid for inaccuracy of predictions in classification problems (problems of identifying which category a particular observation belongs to).

Name a few: log loss, focal loss, exponential loss, hinge loss, relative entropy loss and other.

Note: While more commonly used in regression, the square loss function can be re-written and utilized for classification.

Salgado answered 29/11, 2018 at 14:8 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.