Classification helped us decide which category an instance belongs to. Now we return to a different kind of question: how much?

Regression is the part of supervised learning that predicts numeric outcomes. A city planner might predict commute time. A hospital might predict length of stay. A marketplace might predict demand. A researcher might predict a well-being score from sleep, exercise, stress, and social support.

In this module, we will follow that last story. Imagine a public-health research team studying daily well-being. They collect observations from participants: hours of sleep, minutes of exercise, screen time, stress level, social support, and whether the person worked remotely that day. Their target is a daily well-being score.

The companion script is:

samples/module_03_regression.py

It generates a synthetic well-being dataset and compares linear regression, Ridge, Lasso, Elastic Net, and k-nearest neighbors regression.

Standalone orientation

You can read this article on its own. The only required idea is that supervised learning trains from examples where the answer is already known. Here, the answer is a number, so the task is regression.

If you are reading the whole series, regression is the sibling of classification. Classification predicts “which category?” Regression predicts “how much?” If you are starting here, keep one question in mind throughout the article: how do we learn a numeric prediction from input features, and how do we measure the size of the mistakes?

How to read the examples: X, y, regressors, and residuals

In the regression examples, X is the table of inputs that might help explain daily well-being. It includes sleep_hours, exercise_minutes, screen_hours, stress_level, social_support, and work_mode. Each row describes one simulated day for one participant.

y is the numeric target: wellbeing_score. Unlike classification, where y contains labels such as satisfied or not satisfied, regression uses a continuous or numeric answer. A prediction of 71.4 and a true value of 73.0 are close; a prediction of 44.0 and a true value of 73.0 is a large miss.

The main components in the script are:

When the script calculates MAE, RMSE, and R2, it is comparing predictions to y_test. The difference between each actual value and prediction is a residual. Thinking in residuals keeps regression grounded: every metric is a summary of the model’s mistakes.

The regression promise

A regression model learns a mapping from features to a number:

features -> model -> numeric prediction

For the well-being story, the model might learn that sleep and social support are associated with higher well-being, while stress and excessive screen time are associated with lower well-being. The model does not understand human flourishing. It learns patterns in the measured data.

The simplest regression model is linear regression. It assumes the prediction can be written as a weighted sum of features:

prediction = intercept + weight_1 * feature_1 + weight_2 * feature_2 + ...

This equation is easy to interpret. If the coefficient for sleep is positive, more sleep is associated with a higher predicted score, all else equal. If the coefficient for stress is negative, higher stress is associated with a lower predicted score.

That interpretability is why linear models remain important even in a world full of complex algorithms.

Residuals: where the model meets reality

Regression error is often described through residuals. A residual is the difference between the observed target and the predicted target:

residual = actual value - predicted value

If a participant’s actual well-being score is 72 and the model predicts 68, the residual is 4. If the model predicts 80, the residual is -8.

Residuals are more than arithmetic. They are clues. If residuals are randomly scattered, the model may be capturing the main structure. If residuals are systematically high for low-stress participants or low for remote workers, the model is missing something.

In scikit-learn, training and prediction are familiar:

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

The model learns coefficients that minimize squared residuals on the training data.

Regularization: asking the model to stay humble

Linear regression can become unstable when features are noisy, redundant, or numerous. Regularization solves this by adding a penalty for overly large coefficients.

Ridge regression uses L2 regularization. It discourages large coefficients but usually keeps all features in the model:

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)

Lasso regression uses L1 regularization. It can shrink some coefficients all the way to zero, which makes it useful for feature selection:

from sklearn.linear_model import Lasso
model = Lasso(alpha=0.05)

Elastic Net combines both L1 and L2 regularization:

from sklearn.linear_model import ElasticNet
model = ElasticNet(alpha=0.05, l1_ratio=0.5)

The regularization parameter alpha controls the strength of the penalty. A very small alpha behaves more like ordinary linear regression. A large alpha forces the model to be simpler. The art is choosing enough regularization to improve generalization without erasing real signal.

For beginners, regularization can be understood as a guardrail against overconfidence. For experts, it is a bias-variance control mechanism and a way to stabilize estimation under multicollinearity.

k-nearest neighbors regression: prediction by local memory

k-nearest neighbors can also be used for regression. Instead of voting on a class label, neighboring examples average their numeric targets.

If we want to predict today’s well-being score for a participant, kNN regression finds similar days in the training data and averages their scores. This can capture nonlinear patterns that a straight line misses.

In scikit-learn:

from sklearn.neighbors import KNeighborsRegressor
model = KNeighborsRegressor(n_neighbors=9)

kNN regression is intuitive, but it has the same scaling issue as kNN classification. Features measured on large scales can dominate distance. A pipeline with StandardScaler is usually essential.

Evaluation: the units matter

Regression metrics should be interpreted in the target’s units. If MAE is 4.8 well-being points, we can ask whether that is acceptable for the decision. If RMSE is much larger than MAE, the model may occasionally make large mistakes.

In scikit-learn:

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
predictions = model.predict(X_test)
mae = mean_absolute_error(y_test, predictions)
rmse = np.sqrt(mean_squared_error(y_test, predictions))
r2 = r2_score(y_test, predictions)

R-squared is useful, but it should not be worshiped. It measures improvement relative to predicting the mean. It does not tell you whether a model is fair, causal, stable, or appropriate for intervention.

What to notice when running the sample

The script compares several regressors, but the deeper lesson is how regularization changes the story a model tells. Ordinary linear regression is willing to use whatever coefficients best reduce training error. Ridge regression keeps all features but discourages extreme coefficients. Lasso can drive some coefficients to zero, which means it can act like a feature selector. Elastic Net blends those two behaviors.

Pay attention to the metrics as a set. If MAE and RMSE are close, errors are relatively consistent. If RMSE is much larger than MAE, a few predictions are much worse than the typical prediction. R2 helps describe relative fit, but MAE and RMSE are easier to connect to practical decisions because they use the same units as the target.

The kNN regressor is useful as a contrast. It does not learn coefficients. It predicts from nearby examples. If kNN performs well, the data may contain local patterns that simple linear models do not capture. If it performs poorly, the feature space may be noisy, poorly scaled, or not locally meaningful.

Common regression traps

The most common trap is treating regression as causal. If sleep is associated with well-being in the synthetic data, that does not mean a model has proven that adding one hour of sleep will cause a fixed increase in well-being for every person. Regression learns association from the available features and target.

Another trap is ignoring extrapolation. Linear models can make predictions outside the range of the training data, but those predictions may be unsupported. A model trained on adults sleeping between 4 and 10 hours should not be trusted to explain extreme situations without additional evidence.

Finally, do not let a single aggregate metric hide subgroup behavior. A model can have acceptable overall MAE while performing poorly for high-stress days, remote workers, or people with unusual sleep patterns. Residual analysis is how regression becomes diagnosis instead of just scoring.

The module in one journey

Module 3 deepens the numeric-prediction side of supervised learning. The companion script uses one preprocessing pipeline and several regressors. That pattern matters. When comparing models, keep the data split and preprocessing consistent so the comparison is fair.

The workflow is:

generate data
split features and target
split train and test
fit each pipeline
compare MAE, RMSE, and R2
inspect regularized coefficients

The goal is not to crown one universal winner. Linear regression is interpretable. Ridge is stable. Lasso can simplify. Elastic Net balances two regularization styles. kNN can capture local nonlinear patterns. The best model depends on data, decision, constraints, and evaluation.

The general supervised-learning thread remains intact whether or not you read any earlier article: define X, define y, split the data, fit a pipeline, compare predictions with held-out truth, and interpret the errors in the units of the problem. Regression adds a new responsibility: because predictions are numbers, the size and pattern of each mistake matters.

Run the sample:

python module_03_regression.py