Imagine a neighborhood coffee shop trying to plan for tomorrow. Sometimes the owner wants an exact demand forecast, but sometimes she only needs a category. A normal day needs the usual staffing plan. A high-demand day means ordering more pastries, scheduling another barista, and preparing extra cold brew.

If tomorrow is likely to be a normal day, the usual staffing plan is fine. If tomorrow is likely to be a high-demand day, the shop should order more pastries, schedule another barista, and prepare extra cold brew. The question changes from:

How many orders will we receive?

to:

What kind of day will tomorrow be?

That shift moves us into classification. Classification models predict categories. The category might be high_demand or normal, fraud or legitimate, satisfied or unsatisfied, spam or not_spam. The output is not a continuous number. It is a label.

The companion script for this article is: ML-Blog/module_02_classification.py at main · aduwillie/ML-Blog

It creates a synthetic passenger-satisfaction dataset and compares several scikit-learn classifiers: k-nearest neighbors, logistic regression, Gaussian naive Bayes, and linear discriminant analysis.

Standalone orientation

You can read this article without reading Module 1. The minimum idea you need is this: supervised machine learning uses examples with known answers. The input columns are called X, and the known answers are called y.

This module focuses on cases where y is a category. That category might be satisfied or not satisfied, fraud or legitimate, high demand or normal demand. If you are reading the whole series, this module is the classification branch of the same workflow introduced in Module 1. If you are reading it by itself, treat classification as the art of turning evidence into a class decision while measuring which mistakes matter most.

How to read the examples: X, y, classifiers, and predictions

In this module, X is the table of passenger-experience features. It contains the evidence available before prediction: travel_type, customer_type, flight_distance, delay_minutes, seat_comfort, inflight_service, and online_boarding. Each row represents one passenger trip.

y is the known class label for each row. In the companion script, y = df["satisfied"], where 1 means the passenger was satisfied and 0 means the passenger was not satisfied. Because y contains categories rather than a continuous number, every estimator in this module is a classifier.

The train/test split creates four objects:

X_train  # feature rows used for learning
X_test   # feature rows reserved for evaluation
y_train  # known labels used for learning
y_test   # known labels reserved for evaluation

The preprocessing component converts mixed columns into numeric model input. Numeric columns are imputed and scaled. Categorical columns are imputed and one-hot encoded. The classifier then learns how patterns in X_train relate to labels in y_train.

The predictions are not the same thing as y_test. y_test is the truth we held back. predictions are the model’s guesses. Classification metrics compare those two arrays to answer, “When the model chose a class, how often and in what ways was it right or wrong?”

From a prediction to a decision

Classification begins when a numeric or descriptive situation becomes a decision boundary. For Riverbend, a day with 176 orders may require a different plan from a day with 121 orders. We can create a label:

df["high_demand"] = (df["orders"] >= 175).astype(int)

Now the model is no longer estimating order count. It is estimating whether a day belongs to one class or another.

This framing is powerful, but it should be used carefully. A classification label hides detail. A day with 176 orders and a day with 260 orders both become high_demand, even though they may require different responses. Classification is useful when the decision itself is categorical. If the size of the outcome matters, regression may still be the better tool.

In the companion script, the story changes from coffee demand to passenger satisfaction. Each row is a travel experience. The features describe flight distance, delay, seat comfort, inflight service, travel type, and customer type. The target is whether the passenger was satisfied.

That gives us the familiar supervised learning chain:

passenger experience -> features -> classifier -> satisfied or not satisfied

k-nearest neighbors: learning by similarity

The most intuitive classifier in this module is k-nearest neighbors, often abbreviated as kNN.

Imagine a new passenger. We do not know whether this passenger is satisfied, but we can find similar passengers from the training data. If most of the nearest similar passengers were satisfied, kNN predicts satisfied. If most were unsatisfied, it predicts unsatisfied.

The model’s “story” is simple:

You are likely to behave like examples that look like you.

In scikit-learn:

from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors=7)

kNN is easy to understand, but it is sensitive to feature scale. If flight distance ranges from 100 to 5000 while seat comfort ranges from 1 to 5, distance can dominate the similarity calculation. That is why the sample script places kNN inside a preprocessing pipeline with StandardScaler.

For beginners, kNN is a friendly entry point into classification. For experts, it is a reminder that distance-based models depend heavily on representation. The model is only as good as the geometry created by the features.

Logistic regression: drawing a probabilistic boundary

Despite its name, logistic regression is a classification model. It estimates the probability that an instance belongs to a class.

For passenger satisfaction, logistic regression might learn that long delays reduce the probability of satisfaction, while better seat comfort and inflight service increase it. The model then draws a decision boundary: above a probability threshold, predict satisfied; below it, predict unsatisfied.

In scikit-learn:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=1000)

Logistic regression is often a strong first serious classifier. It trains quickly, works well with many tabular datasets, produces probabilities, and can be interpreted through coefficients. It is not always the most accurate model, but it is frequently the most useful starting point.

The important conceptual move is that classification can be probabilistic. A model can say, “I estimate a 0.82 probability of satisfaction,” not merely “satisfied.” Probabilities let us adjust thresholds to match business needs. If missing an unsatisfied customer is costly, we may choose a different threshold than the default 0.5.

Naive Bayes: using evidence quickly

Naive Bayes classifiers are based on Bayes’ theorem. They combine evidence from features to estimate the likelihood of each class. The word “naive” refers to a simplifying assumption: features are treated as conditionally independent given the class.

That assumption is often false. Seat comfort and inflight service may be related. Delay and travel type may interact. Yet naive Bayes can still work surprisingly well, especially when the feature representation is appropriate and speed matters.

In scikit-learn:

from sklearn.naive_bayes import GaussianNB
model = GaussianNB()

Gaussian naive Bayes assumes numeric features follow class-specific normal distributions. It is not always the best model for modern tabular work, but it teaches an important idea: classification can be framed as comparing how likely the observed evidence is under each class.

The expert lesson is not that the naive assumption is always acceptable. The expert lesson is that simple probabilistic models can be valuable baselines, especially when you need speed, interpretability, or a sanity check against more complex methods.

Discriminant analysis: modeling class separation

Linear discriminant analysis, or LDA, tries to find linear combinations of features that separate classes well. It can be used as a classifier and, in some settings, as a supervised dimensionality reduction method.

In scikit-learn:

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
model = LinearDiscriminantAnalysis()

LDA works best when its assumptions are reasonably aligned with the data. It assumes classes have Gaussian-like feature distributions and similar covariance structure. When those assumptions are not badly violated, LDA can be effective and efficient.

For our passenger-satisfaction story, LDA asks: can we project the customer experience into a space where satisfied and unsatisfied passengers are cleanly separated?

This is a useful mental model. Many classifiers are trying, in different ways, to separate classes. kNN separates by local neighborhoods. Logistic regression separates with a learned boundary. Naive Bayes separates by comparing likelihoods. LDA separates by modeling class distributions.

Evaluation: accuracy is only the first page

Classification evaluation starts with a confusion matrix. It counts correct and incorrect predictions by class:

true positives
true negatives
false positives
false negatives

From these counts we derive metrics. Accuracy measures the fraction of predictions that are correct. It is easy to understand, but it can mislead when classes are imbalanced. If only 5 percent of transactions are fraudulent, a model that always predicts “not fraud” is 95 percent accurate and completely useless.

Precision asks: when the model predicts positive, how often is it right? Recall asks: of all actual positives, how many did the model find? F1-score balances precision and recall.

In scikit-learn:

from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test, predictions))
print(classification_report(y_test, predictions))

The metric should match the decision. If Riverbend is deciding whether to schedule one extra person, false positives may cost money. If a hospital is screening for disease, false negatives may be far more serious. Classification is never just about choosing the highest accuracy number.

What to notice when running the sample

The companion script trains four classifiers on the same X_train, X_test, y_train, and y_test. That consistency matters. If the data split changed for each model, the comparison would be noisy and unfair. By keeping the split and preprocessing strategy stable, the script lets you focus on how the model families behave differently.

Look at the confusion matrix before reading the classification report. The confusion matrix tells the concrete story: how many satisfied passengers were missed, how many unsatisfied passengers were incorrectly called satisfied, and how many examples landed in each correct cell. The classification report then turns those counts into precision, recall, and F1-score.

Also notice that kNN, logistic regression, Gaussian naive Bayes, and LDA all receive the same processed features, but they interpret evidence differently. kNN asks, “Who are this passenger’s neighbors?” Logistic regression asks, “What boundary separates the classes?” Naive Bayes asks, “Which class makes this evidence more likely?” LDA asks, “Can the classes be separated by a discriminant direction?” That is why model comparison is not only a scoreboard; it is a comparison of assumptions.

Common classification traps

The first trap is trusting accuracy without checking class balance. If one class dominates, accuracy can look strong while the model ignores the minority class. The second trap is treating the default probability threshold as sacred. A classifier may output probabilities, but the decision threshold should reflect the cost of false positives and false negatives. The third trap is forgetting that labels are human definitions. If “satisfied” is collected from a biased survey process, the model learns that process, not pure satisfaction.

Classification is powerful because it turns messy situations into decisions. It is risky for the same reason. The boundary you define becomes the world the model learns.

The module in one journey

Module 2 turns machine learning from “predict a number” into “choose a class.” The same foundation remains: define the target, prepare the features, split the data, train the model, evaluate on held-out examples, and compare models with metrics that reflect the real decision.

kNN teaches similarity. Logistic regression teaches probabilistic boundaries. Naive Bayes teaches evidence. Discriminant analysis teaches class separation. Together, they show that classification is not one algorithm. It is a family of ways to answer the same kind of question:

Given what we know about this instance, which category is most likely?

Run the companion script to see the full comparison:

python module_02_classification.py