Statistical learning refers, in broad terms, to a collection of methods through which one attempts to estimate an unknown function from data. Following the formulation used by James, Witten, Hastie, and Tibshirani, we may imagine a response variable \(Y\) related to a set of input variables \(X\) through:

where \(f\) represents an unknown systematic relationship and \(\varepsilon\) collects variation not explained by the model. The problem is therefore to obtain from the available observations an estimate \(\hat f\) sufficiently useful for the purpose of the investigation.

That purpose may be predictive, inferential, or descriptive. In some applications the primary concern is predicting new observations; in others it is understanding which variables are related to the response and in what manner; and in many investigations both aims appear simultaneously.

1. Statistical learning and machine learning

Hastie, Tibshirani, and Friedman use the term statistical learning to describe a broad collection of methods for data analysis, including prediction, inference, classification, variable selection, regularization, clustering, and other forms of structure extraction.

Machine learning developed historically under a strong influence from computer science and algorithm design. Statistics, by contrast, developed a tradition particularly concerned with probability models, estimation, uncertainty, and inference.

This historical difference helps explain certain emphases, but does not constitute a rigid logical boundary. Statistics also builds predictors, while machine learning also uses probabilistic models, estimation, regularization, and inference. The two traditions have increasingly converged because they frequently address the same underlying problems.

Nor is it correct to say that machine-learning methods make no assumptions. Many impose fewer explicit parametric assumptions about the distribution of the data, but they still embody an inductive bias through the allowed function class, algorithmic architecture, loss function, regularization, invariances, or representation.

In this context, a hypothesis space is a set of candidate functions or rules from which a learning procedure may select:

This should not be confused with the hypotheses \(H_0\) and \(H_1\) of classical Neyman–Pearson hypothesis testing. There is a broad logical analogy, but the mathematical objects and inferential problems are different.

2. Learning paradigms

Online learning belongs to another classificatory dimension. It refers to procedures that update the model as new observations arrive. One may therefore have supervised online learning, unsupervised online learning, and so forth.

Unsupervised methods include clustering, dimensionality reduction, anomaly detection, density estimation, and representation learning, among others.

3. Positives and negatives in binary classification

• True positive (TP): the case is positive and the model predicts positive.
• False positive (FP): the case is negative and the model predicts positive.
• True negative (TN): the case is negative and the model predicts negative.
• False negative (FN): the case is positive and the model predicts negative.

In medicine, for example, “positive” might represent the presence of a disease. A false positive would classify a healthy person as diseased, while a false negative would fail to detect the disease in someone who has it.

4. The confusion matrix

The main diagonal contains correct classifications; the off-diagonal cells contain errors.

A confusion matrix can be computed on training data, but also on validation data, test data, or genuinely new external observations. This distinction is fundamental when the objective is to assess generalization.

5. Classification metrics

5.1. Accuracy

The corresponding classification error is:

Accuracy may be misleading under strong class imbalance. A classifier that always predicts a 99% majority class can achieve 99% accuracy while learning virtually nothing about the minority class.

5.2. Sensitivity or recall

This is the proportion of actual positives correctly identified.

5.3. Specificity

This is the proportion of actual negatives correctly identified.

5.4. Precision or positive predictive value

In diagnostic terminology this quantity is commonly called the positive predictive value.

5.5. Negative predictive value

5.6. Summary

6. Training error and generalization error

If \(\hat f\) is a classifier estimated from \(n\) training observations, its training error is:

This measures how well the procedure reproduces the observations used to fit it, but does not necessarily answer the central predictive question: what happens on new data?

Here \((X_0,Y_0)\) denotes a new observation not used during training.

Validation sets, test sets, cross-validation, and related procedures are designed to estimate this out-of-sample performance.

An extremely small training error may even indicate overfitting when performance deteriorates substantially on new data.

7. Statistics and machine learning

The relationship between statistics and machine learning cannot be reduced to a simple opposition. Both fields employ functions, optimization, probability, geometry, regularization, and computation in order to extract structure from data.

One historical difference concerns the degree to which functional form is specified explicitly before parameter estimation.

If theory or evidence suggests a nonlinear relationship with \(X_2\), we may instead specify:

The relationship is nonlinear in \(X_2\), yet the model remains linear in the parameters.

Decision trees, random forests, support-vector machines, and neural networks use different function classes and can represent highly nonlinear relationships without requiring the analyst to write a particular polynomial equation in advance.

This does not mean that such methods are “structure-free.” Their structure is embodied in the hypothesis class, algorithmic architecture, optimization criterion, and regularization.

8. Generalized linear models as supervised learning

Generalized linear models provide a particularly clear example of the overlap between statistics and machine learning.

Historically and theoretically, they are statistical models. They specify a response distribution, a linear predictor:

and a link function:

Yet when a GLM is fitted to pairs:

for the purpose of predicting \(Y\) for new values of \(X\), it is also functioning as a supervised-learning method.

Bayesian treatment is not what creates this status. A GLM may be used for supervised learning under frequentist, Bayesian, or penalized estimation. What matters is the learning problem and how the model is used.

8.1. Logistic regression is still regression

The object being modeled is a conditional probability. Logistic regression is therefore unequivocally a regression model.

At the same time, an estimated probability can legitimately be converted into a classification rule:

8.2. The elementary criterion for supervision

In the simplest classification, a problem is supervised when training data contain paired inputs and targets:

The \(y_i\) values provide the signal against which predictions can be assessed.

In an unsupervised problem we instead observe:

without a target variable associated with each observation. Clustering is one possibility in this setting, but by no means the only one.

Statistical learning theory occupies precisely this common territory between statistics, mathematics, and computation. Its purpose is not to replace one discipline with another, but to study systematically how generalizable information can be extracted from data.

The quality of learning cannot be assessed solely by asking how well a model reproduces the data used to construct it. The decisive question is whether the learned structure represents a sufficiently stable regularity to retain validity when confronted with new information.