Let me explain this with an example,
Imagine a kid sitting by a fire to keep warm, but not knowing the danger of fire, he puts his finger into it and gets burned. The next time he sits by the fire, he doesn't get burned, but he sits too close and gets too hot and moves away from the fire. The third time he sits by the fire, he finds the distance that keeps him warm without exposing him to any danger. In other words, he learns from his experience that is getting burned and then getting too hot. The kid learns the optimal distance to sit from the fire. The heat from the fire in the example acts as cost function, it helps the learner to correct himself to minimize mistakes.
A cost function is a measure of how wrong the model is in terms of its ability to estimate the relationship between x and y. This is typically expressed as a difference or distance between the predicted value and the actual value. The cost function can be estimated by iteratively running the model to compare estimated predictions against “ground truth”, the known values of y.
We can measure the accuracy of our hypothesis function by using a cost function. This takes an average difference (actually a fancier version of an average) of all the results of the hypothesis with inputs from x’s and the actual output y’s.
Consider a scenario where we wish to classify data. Suppose we have the height & weight details of some cats & dogs. Let us use these 2 features to classify them correctly. If we plot these records, we get the following scatter plot:
Cost Function Scatter plot Fig 1: Scatter plot for height & weight of various dogs & cats. Blue dots are cats & red dots are dogs. Following are some solutions to the above classification problem.
Probable solutions Fig: Probable solutions to our classification problem.
Essentially all three classifiers have very high accuracy but the third solution is the best because it does not misclassify any point. The reason why it classifies all the points perfectly is that the line is almost exactly in between the two groups, and not closer to any one of the groups. This is where the concept of cost function comes in. Cost function helps us reach the optimal solution. The cost function is the technique of evaluating “the performance of our algorithm/model”.
It takes both predicted outputs by the model and actual outputs and calculates how much wrong the model was in its prediction. It outputs a higher number if our predictions differ a lot from the actual values. As we tune our model to improve the predictions, the cost function acts as an indicator of how the model has improved. This is essentially an optimization problem. The optimization strategies always aim at “minimizing the cost function”.
Types of Cost function:
There are many cost functions in machine learning and each has its use cases depending on whether it is a regression problem or classification problem.
Regression cost function
Binary classification cost functions
Multi-class classification cost functions