lassence.net

Field notes from the AI frontier

Performance metrics of ML models

--- date: · theme: data · words: 477 ---

For classification models

  1. Confusion matrix: not a metric per se, but a tabular summary that is commonly used to evaluate the performance of a classification model. It provides a breakdown of the model's predictions versus the actual ground truth labels.
Predicted 0Predicted 1
Actual 0True NegativesFalse Positives
Actual 1False NegativesTrue Positives
  1. Accuracy: simplest metric, measures the fraction of correct predictions made by the model over the total number of predictions.
Accuracy=Number of correct predictionsTotal number of predictions \text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}}
  1. Precision: measures the fraction of true positive predictions out of all positive predictions made by the model. It addresses the question: "Of all the instances predicted as positive, how many were actually positive?"
Precision=True PositivesTrue Positives+False Positives \text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}
  1. Recall: measures the fraction of true positive predictions out of all actual positive instances. It answers: "Of all the actual positive instances, how many were correctly predicted as positive?"
Recall=True PositivesTrue Positives+False Negatives \text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}
  1. F1-score: harmonic mean of precision and recall, the F1-score combines them into a single metric for better interpretation. Useful when you want to balance the trade-off between precision and recall, and when both of these metrics are equally important for your problem. Also, it is appropriate for imbalanced datasets.
F1-score=2×Precision×RecallPrecision+Recall \text{F1-score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
  1. Area Under the ROC Curve (AUC-ROC): For binary classification problems, the ROC curve plots the true positive rate against the false positive rate at various classification thresholds. The AUC-ROC measures the area under this curve, with a higher value indicating better performance.
  1. Log Loss (Cross-Entropy Loss): A probabilistic metric that measures the performance of a classification model whose output is a probability value between 0 and 1. It penalizes confident and incorrect predictions.
Log Loss=−1N∑i=1N[yilog⁡(pi)+(1−yi)log⁡(1−pi)] \text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right]

For regression models

  1. Mean Squared Error (MSE): measures the average squared difference between the predicted values and the actual values.
MSE=1N∑i=1N(yi−y^i)2 \text{MSE} = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2
  1. Root Mean Squared Error (RMSE): Square Root of the MSE, easier to interpret since it’s the same order of magnitude than the dependent variable.
RMSE=1N∑i=1N(yi−y^i)2 \text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2}
  1. Mean Absolute Error (MAE): measures the average absolute difference between the predicted values and the actual values. MAE is less sensitive to outliers compared to RMSE because it does not square the errors.
MAE=1N∑i=1N∣yi−y^i∣ {MAE} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i|
  1. R-squared (R²): measures the proportion of variance in the target variable (TSS, aka Total Sum of Squares) that is explained by the independent variables in the model (RSS, aka Residuals Sum of Squares).
R2=1−RSSTSS=1−∑i=1N(yi−y^i)2∑i=1N(yi−yˉ)2 R^2 = 1 - \frac{RSS}{TSS} = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2}