Model quality
The quality of a model can be determined by the comparison of the predicted values with the original values. There are two model types: regression model and classifier. Depending on the model type the errors are calculated in different ways.
Regression models:
Mostly the absolute distance between the forecast and the original value is calculated and afterwards either added and averaged (mean absolute error) or squared, added and averaged (mean square error). The advantage of the mean absolute error is its better readability. The mean square error, however, is better to use in analyses (derivation). Also a look at a scatter chart that visualizes the opposition of predicted values and original values can be helpful.
Classifiers:
Here the classification error can be determined. However, it is often the case that one class is much underrepresented. As a consequence, a low classification error could be achieved because the class which occurs most often is always predicted. To work against such a development a confusion matrix can be established.
Prediction\Original class | Class A | Class B |
---|---|---|
Class A | Amount prediction A right | Amount prediction B false |
Class B | Amount prediction A false | Amount prediction B right |
From this one can derive different quality criteria. For example, the sensitivitiy for Class A can be calculated: Amount prediction A right /Amount appearance A. I.e., which is the amount of correctly predicted objects of one class. The correctness, however, is Amount prediction A right/ Amount prediction A. I.e., which is the amount of correct prediction of one class compared with all predictions of this class. There are even more ways of deriving quality criteria from a confusion matrix (see https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers). Depending on the focus (each appearance of a class must be found/ the prediction of a class always has to be right) one of the measures can be used to optimize classifiers.