The expensive habit hiding inside AI enthusiasm

The easiest mistake to make in the current AI cycle is to treat every automation, prediction, classification, ranking, search, routing, extraction, and decision-support problem as a large language model problem.

It is an understandable mistake. LLMs are astonishingly flexible. They can summarize, classify, explain, translate, write code, reason over messy text, generate structured output, and talk to users in natural language. They turn ambiguous inputs into plausible answers with very little upfront feature engineering. For a prototype, that feels magical.

But production systems do not run on magic. They run on latency budgets, failure modes, unit economics, deployment constraints, observability, privacy boundaries, and scale. A model that feels cheap during a demo can become expensive when it sits on a hot path. A model that feels intelligent in a notebook can become brittle when downstream systems expect deterministic output. A model that feels easy because it avoids feature engineering can push complexity into prompts, retries, evaluations, vendor dependencies, and incident response.

The deeper point is not that LLMs are bad. The point is that LLMs are not the default answer to every intelligence problem. Many of the highest-return problems in software still need simpler tools: rules, heuristics, linear models, tree models, nearest-neighbour search, anomaly detectors, logistic regression, gradient-boosted trees, small embedding models, or even a well-designed SQL query.

The biggest wins often come from matching the problem to the smallest model that can solve it reliably.

The old lesson still applies: start simple

Google’s practical machine learning guidance says something that still feels almost subversive in the LLM era: most gains come from great features and solid engineering, not from fancy algorithms. The recommended first step is not to reach for the most advanced model. It is to build a reliable pipeline, define a reasonable objective, add common-sense features, and keep the system solid end to end.

That advice is more relevant now, not less. LLMs make it easier to skip the boring parts. They let teams jump straight to an impressive demo before answering the basic questions:

1. What is the actual decision being made?
2. What inputs are available at serving time?
3. What does “good” mean?
4. How often does the answer need to change?
5. What happens when the model is wrong?
6. Does the system need language generation, or just a prediction?
7. Can we solve 80 percent of the value with a cheaper, smaller, more predictable method?

The danger is not that a team uses an LLM. The danger is that the team uses an LLM before understanding the shape of the problem.

LLMs are general-purpose engines, but many problems are narrow

An LLM is like a helicopter. If you need to cross a mountain range, evacuate someone from a remote area, or get a broad aerial view, it is an extraordinary tool. But if you need to move groceries three blocks, a bicycle is faster, cheaper, quieter, easier to maintain, and less likely to wake the neighbourhood.

Many business and product problems are three-block grocery problems.

A spam filter does not always need a frontier model. A churn-risk score often does not need generative reasoning. A routing decision might be a small classifier. A recommendation candidate filter might be a gradient-boosted tree. A support-ticket priority score might start with rules and logistic regression. A device-side sensor classifier might run on a microcontroller with kilobytes of memory.

The job is not to prove that the largest model can solve the task. The job is to build the smallest reliable system that meets the product, cost, latency, privacy, and maintenance constraints.

The hidden cost of “just call the LLM”

LLMs shift cost from training and feature work into inference, evaluation, reliability, and operations.

Chip Huyen’s writing on LLM engineering makes this trade-off concrete: prompt engineering can be cheap and fast for experimentation, but the cost of LLMOps is in inference. A long prompt plus generated output may be acceptable for dozens of examples, but it can become painful at millions or billions of predictions. The same analysis also highlights latency and consistency issues: generated output is sequential, APIs add variance, and natural-language interfaces can fail silently when prompts change.

This matters because many production decisions are high-volume and low-margin. A model that runs once per user per week has a very different cost profile from a model that runs on every search result, every feed item, every checkout attempt, every IoT reading, or every support message.

There are four costs teams often underestimate.

Simple ML models are not literally free. They have development, training, monitoring, and maintenance costs. But once trained and deployed, their marginal inference cost can be close to zero relative to API-based LLM calls. A logistic regression model, a small random forest, or a compact gradient-boosted tree can run inside an existing application process, on a CPU, without a GPU, without a network call, and without paying per token.

At scale, “almost free per prediction” is not a small detail. It can be the difference between a feature that can run everywhere and a feature that only works for premium paths.

Simple models can run where LLMs cannot

One of the strongest arguments for simple ML is deployment freedom.

ONNX Runtime describes itself as a cross-platform machine-learning model accelerator that can run models from frameworks such as PyTorch, TensorFlow/Keras, TFLite, and scikit-learn, with support across hardware and operating systems. LiteRT for Microcontrollers is even more extreme: its microcontroller runtime is designed for devices with only a few kilobytes of memory, with a core runtime that fits in 16 KB on an Arm Cortex-M3 and runs without an operating system, standard C or C++ libraries, or dynamic memory allocation.

That is a different universe from serving a large language model.

Simple models can run:

1. Inside a web service on commodity CPU.
2. In a mobile app without a network call.
3. In a browser with WebAssembly.
4. On an embedded Linux device.
5. On a microcontroller close to a sensor.
6. In a serverless function with small memory limits.
7. In a batch job over millions of records.

That changes the architecture. If inference can run locally, the system can be faster, cheaper, more private, and more resilient. There is no prompt to transmit, no external API latency, no token bill, and no dependency on a large model’s availability.

This is especially important under scale. A simple fraud score that runs in 2 milliseconds on CPU can be called everywhere. An LLM-based fraud explanation that takes seconds and costs money per request might still be valuable, but probably not on every transaction. The architecture should use each model where it fits.

The decision is not LLM versus no AI

The debate is often framed incorrectly. The question is not, “Should we use AI or avoid AI?” The better question is, “What level of intelligence does this part of the system actually need?”

There is a ladder of solutions:

Manual process  
  -> rule or heuristic  
  -> SQL query or deterministic program  
  -> simple statistical model  
  -> tree-based or ranking model  
  -> embedding model or vector search  
  -> small task-specific neural model  
  -> LLM for language-heavy reasoning or generation 
  -> agentic workflow with tools and orchestration

The higher you climb, the more flexibility you get. But you also tend to add cost, latency, evaluation burden, operational complexity, and failure modes. A mature AI strategy does not climb the ladder automatically. It climbs only when the problem demands it.

What simple ML is still excellent at

Simple ML models are not outdated. They are still extremely strong for many structured, repeated, measurable tasks.

Classification

If the task is to assign one of a small number of labels, a simple classifier should often be the first serious baseline. Examples include spam detection, support-ticket routing, lead scoring, policy violation flags, customer segment prediction, retry risk, and document-type classification.

An LLM may help bootstrap labels, explain edge cases, or handle free-form ambiguity. But the production classifier may not need to generate language at all.

Ranking and scoring

Search ranking, recommendation candidate scoring, notification prioritization, and risk scoring often benefit from compact models that combine behavioral, contextual, and historical features. These models can be fast enough to run on many candidates per request.

An LLM may be useful for semantic understanding or query rewriting, but it is usually too expensive to score thousands of items one by one in a hot path.

Forecasting and anomaly detection

Many operational problems need trend detection, seasonality, thresholds, or anomaly scores. A classic time-series model or statistical detector can be easier to monitor and explain than an LLM. If a disk metric, payment failure rate, or conversion funnel step drifts, the first question is rarely, “What would a language model say?” It is, “Is this statistically unusual, and what changed?”

Extraction from regular formats

Not every extraction task needs generative AI. If the input is semi-structured, stable, or repetitive, regular expressions, parsers, templates, OCR plus rules, or a small classifier may be enough. LLMs become more attractive when the format varies widely, the language is messy, or the system needs to infer intent across context.

Edge and device-side intelligence

Wake-word detection, vibration classification, sensor anomaly detection, gesture recognition, and simple image or audio classification often need to run near the device. In these cases, the constraints are severe: power, memory, latency, network availability, and privacy. Tiny models are not a compromise; they are the only practical architecture.

A practical routing example

Imagine a customer-support system that routes incoming tickets. The LLM-first design might look like this:

Ticket text 
  -> LLM prompt 
  -> department, urgency, summary, suggested answer

This is attractive because one call does many things. But it mixes multiple tasks with different requirements:

1. Department routing needs a stable label.
2. Urgency scoring needs calibrated risk.
3. Summarization needs natural language.
4. Suggested response needs generation and policy review.

An alternative architecture separates the tasks:

Ticket text  
  -> lightweight classifier for department  
  -> rules + model score for urgency  
  -> LLM only when a human needs a summary or draft response

This design is less flashy, but it is usually better. The cheap model handles the high-volume, repetitive decisions. The LLM is reserved for the language-heavy work where generation matters.

The result is lower cost, lower latency, clearer monitoring, and better failure isolation. If the summarization path degrades, routing can still work. If the classifier needs retraining, response drafting does not have to change. Each part of the system has the right tool for its job.

The right architecture is often a cascade

The best production systems often use cascades rather than one giant model.

A cascade is a sequence of increasingly expensive steps. Cheap logic handles the easy cases. More expensive models handle ambiguous cases. Humans or expert workflows handle the rare high-risk cases.

Input  
  -> deterministic validation  
  -> heuristic or simple model  
  -> confidence threshold  
  -> LLM for ambiguous/high-value cases  
  -> human review for low-confidence/high-risk cases

This pattern respects scale. If 70 percent of requests are easy, do not pay LLM prices for 100 percent of them. If 20 percent need semantic understanding, use a stronger model there. If 1 percent are risky, route them to people or stricter review.

The cascade also creates better observability. You can track how many cases are solved by each layer, where confidence drops, where errors cluster, and which part of the system deserves investment.

When an LLM is the right answer

The point of this argument is not to avoid LLMs. Some problems genuinely are LLM problems.

Use an LLM when the task needs open-ended language understanding or generation, when input variation is high, when the schema cannot capture the nuance, or when the value of a better answer justifies the cost.

LLMs are especially useful for:

1. Summarizing long, messy, natural-language context.
2. Drafting responses, documents, or explanations.
3. Translating ambiguous user intent into structured actions.
4. Extracting information from highly variable text.
5. Conversational interfaces where language is the product surface.
6. Reasoning across heterogeneous documents.
7. Assisting humans in low-volume, high-value workflows.

The key phrase is “justifies the cost.” Cost is not only dollars. It includes latency, reliability, privacy, evaluation complexity, and user trust.

The evaluation burden is different

Simple models and LLMs need different evaluation strategies.

For a traditional classifier, you can often build a labeled test set and measure precision, recall, calibration, confusion matrices, and drift. The model’s output space is constrained. That does not make evaluation trivial, but it makes the contract clearer.

For LLMs, the output space is much wider. You may need golden examples, rubric-based evaluation, behavioural tests, adversarial prompts, human review, format validation, safety checks, latency tracking, cost tracking, and regression tests across prompt versions. Microsoft’s CheckList paper is a useful reminder that accuracy alone is not enough for NLP systems; behavioural testing matters because models can pass aggregate metrics while failing important capabilities or invariance tests.

This does not mean LLMs are impossible to evaluate. It means teams should not pretend that “the demo looked good” is a production evaluation.

The broader rule is simple: the more flexible the model, the more serious the evaluation needs to be.

The privacy and data-boundary argument

Simple models also change the privacy posture of a system. If a model can run locally, sensitive data may not need to leave the device, tenant, region, or service boundary. LiteRT’s microcontroller documentation explicitly calls out privacy as a benefit of on-device intelligence: data can stay on the device rather than depending on reliable internet connections or remote inference.

This matters for enterprise software, healthcare, finance, government, industrial systems, and consumer devices. Sending everything to an LLM endpoint may be unacceptable, expensive, slow, or simply unnecessary. A small on-device model that detects a local pattern can be more privacy-preserving than a powerful remote model that sees every raw event.

Privacy is not just a compliance concern. It is an architecture constraint.

The scale argument: hot path versus cold path

One of the most useful design questions is whether the model sits on a hot path or a cold path. A hot path runs frequently, synchronously, or in direct user experience. Examples include ranking, routing, fraud checks, content filtering, personalization, autocomplete, checkout decisions, and device control. Hot paths usually demand low latency, high reliability, predictable cost, and graceful degradation.

A cold path runs less frequently, asynchronously, or as assistance to humans. Examples include weekly insight reports, internal analysis, support-agent summaries, onboarding document drafts, policy review assistance, and exploratory search across documents. Cold paths can often tolerate higher latency and cost because the value per call is higher.

LLMs often make more sense on cold paths first. Simple ML often dominates hot paths. That division is not absolute, but it is a powerful default.

A decision framework: choose the smallest sufficient model

Before reaching for an LLM, ask these questions:

The answer may still be “use an LLM.” But the decision should be earned, not assumed.

The hidden organizational problem

LLM overuse is not only a technical problem. It is an organizational problem. Teams reach for LLMs because they reduce upfront coordination. A prompt can absorb ambiguity that would otherwise require product clarity, data modelling, feature design, instrumentation, or cross-functional agreement. That makes LLMs feel like a shortcut. But ambiguity does not disappear; It moves.

If the team does not define labels, the prompt has to guess. If the team does not define success, evaluation becomes vibes. If the team does not understand failure modes, users discover them. If the team does not design escalation paths, the model becomes the escalation path.

Simple ML forces useful discipline. It asks for labels, features, metrics, thresholds, and feedback loops. That discipline can feel slower at the beginning, but it often produces a system that is cheaper, clearer, and easier to improve. The best teams bring that discipline to LLM systems too.

A balanced AI architecture

The most robust AI products will not be “LLM everywhere” products. They will be systems that compose multiple kinds of intelligence.

They might use:

1. Rules for hard constraints.
2. SQL for factual retrieval.
3. Simple classifiers for stable labels.
4. Ranking models for high-volume scoring.
5. Embeddings for semantic retrieval.
6. LLMs for reasoning, summarization, drafting, and interface flexibility.
7. Human review for high-risk or low-confidence cases.

This is not less ambitious than using an LLM everywhere. It is more ambitious because it treats AI as system design, not model selection.

The goal is not to maximize model size. The goal is to maximize product value per unit of complexity.

What leaders should ask before funding another LLM feature

Leaders do not need to become model experts, but they should ask better questions:

1. What is the simplest baseline?
2. What is the expected volume?
3. What is the cost per thousand, million, and billion decisions?
4. What latency budget does the user experience require?
5. What happens when the model is unavailable?
6. What data leaves our boundary?
7. How will we evaluate quality before and after launch?
8. Can a small model handle the common cases?
9. Can the LLM be reserved for ambiguous or high-value cases?
10. What will we monitor in production?

These questions prevent AI strategy from becoming model fashion.

What builders should do next

If you are building an AI feature, start with a model ladder rather than a model preference.

Write down the task. Define the output. Estimate the volume. Identify the latency budget. Decide whether the output must be generated language or a structured prediction. Build the simplest baseline. Measure it. Only then decide whether a larger model is justified.

That process may sound slower than prompting an LLM, but it often moves faster in the long run. It creates a baseline, exposes the real difficulty, and prevents a prototype from becoming a cost center.

The mature AI builder is not the one who uses the biggest model. It is the one who knows when not to.

The bigger story

The industry does not need fewer LLMs. It needs better judgment about where LLMs belong.

LLMs are extraordinary when the problem is language-heavy, ambiguous, contextual, or generative. But many valuable problems are narrow, repeated, measurable, and scale-sensitive. For those problems, simple ML is not a fallback. It is often the right architecture.

The next wave of AI maturity will come from restraint. Not anti-LLM restraint, but systems restraint: choosing the smallest sufficient model, putting expensive reasoning where it creates real value, and letting cheap, fast, reliable models do the work they are already good at.

Not everything is an LLM problem.

And that is good news, because some of the best AI systems will be the ones that know exactly when a large language model is unnecessary.