AI Problems Are Search Problems
An LLM doesn't know anything about your business. So everything a workflow or an agent do depends on the context you give it. Handing it the right stuff, out of thousands of possible things, is a search problem — no matter what part of the system is doing the handing.
A stronger model may reason better over the data in front of it. That's real, and every release makes it more real. But it doesn't help you when the relevant document never made it into the window, and it doesn't help much when the document is in there buried under forty others that looked vaguely similar. In the first case the model is guessing. In the second it's paying full price per token to go through the entire pile, and getting worse at the task while it does.
Which is why I keep landing on the same claim: most of what we call AI problems are search and relevance problems in a costume.
Bigger windows are not the same as better windows
Context windows grow every year, and every time they do somebody declares retrieval dead. Just put everything in.
The first objection is cost and latency, which is obvious and which people accept surprisingly quickly. The second one is the interesting one: models do not use a long context evenly.
Lost in the Middle is where this started — accuracy is highest when the relevant information sits at the beginning or the end of the input and sags when it's buried in the middle. Chroma's Context Rot took the same question across 18 frontier models and found the degradation shows up well before the advertised limit, on tasks simple enough that it shouldn't.
And it gets worse exactly where retrieval matters most. The classic needle-in-a-haystack test hides a sentence that shares vocabulary with the question, which turns it into a lexical match. NoLiMa rebuilt the test so the needle and the question have almost no words in common — you have to actually infer the connection. At 32K tokens, 11 of the 12 long-context models they tested had fallen below half their own short-context score.
Which is the whole game, isn't it. Users don't ask questions using your documents' vocabulary. That's why we needed search in the first place.
So the window is a budget. Not a big one, and not one where every slot is worth the same. Deciding what goes in it is a ranking decision made under a size constraint, on every single call. That's a search problem, and it doesn't go away at 2M tokens.
Five things that are search and don't look like it
1. Document selection
The obvious one, and still the one that goes wrong most. An insurance assistant answering "why was my claim denied". A support agent reading a customer's ticket history. A legal tool over policy documents.
Everyone knows this part is retrieval. Where it falls over is the assumption that owning a vector database means retrieval is handled — chunk, embed, top-5, ship. Then someone asks about a specific exclusion clause, and that clause got split down the middle of two chunks, or ranked eighth. The model answers confidently from the seven chunks that did make it, and it sounds great.
As a search problem: given this question, find the passages most likely to contain the answer — and know how often you don't.
That second half is the part that gets skipped. If you don't know your recall, you can't tell whether a bad answer is a model problem or a retrieval problem, and you will spend three weeks rewriting a prompt to fix a chunking bug.
2. Tool selection
A dozen tools fit comfortably in a prompt. The model reads every description, picks the right one, and nobody files it under retrieval.
Three hundred tools do not fit. Every description sits in context on every call, near-duplicates pile up — get_user, fetch_user_profile, lookup_customer — and accuracy drops exactly where you'd expect. The fix is to stop sending all of them: index the descriptions, retrieve the handful that match the request, pass only those. Some providers now do this for you server-side with BM25 over the tool descriptions — the same ranking function from the hybrid search post, pointed at tools instead of products.
As a search problem: given this request, find the tools whose descriptions best match what the user is trying to do.
Which is a funny place to end up: your agent is now running a retrieval system over its own capabilities, with all the usual failure modes attached — including the one where the user names a tool explicitly and a vector search talks the model out of it.
3. Agent memory
"Memory" is a friendly word for a hard problem. The agent accumulates things across sessions — preferences, past decisions, corrections the user made — and eventually there are too many to hand over wholesale.
So you hand over some. Which ones? The ones relevant to this turn. That's retrieval, over a corpus that happens to be small, personal, self-written, and full of near-duplicates and stale entries quietly contradicting each other.
As a search problem: given the current turn, find the memories worth spending context on — and skip the ones that have been superseded.
Memory systems fail in ways a search person recognizes on sight. No dedup, so the same fact is stored nine times in nine phrasings and outvotes everything else. No recency weighting, so a preference the user changed in March still outranks the correction. No eval, so nobody finds out.
4. Intent classification
You have an incoming message and a set of routes — billing, technical, sales, cancellation. The standard move is to ask an LLM to classify it, or to fine-tune a small classifier.
The retrieval version: embed the message, find the nearest labeled examples, route by what the neighbors say. It isn't the right answer everywhere, but it has properties the other two don't. Adding a category means adding examples, not rewriting a prompt or retraining anything. Fixing one specific misroute means adding that one example, and it stays fixed. And you can look at the neighbors that produced the decision, which beats "the model said billing" when someone asks why.
As a search problem: given this message, find the most similar messages we've already labeled.
Worth noticing that this is question extraction pointed at a different target — matching intent to intent instead of intent to content.
5. Eval set curation
Least obvious of the five, and the one that pays off quietly for years.
You have fifty thousand logged conversations and you need an eval set. Not fifty thousand — a few hundred that cover what actually varies. Sample randomly and you get three hundred flavors of the same easy request, plus a false sense of coverage.
What you want is spread: cluster the traffic, sample across clusters, and deliberately pull the cases sitting nearest to known failures. And when a bug comes in, the first question is "how many other conversations look like this one" — which is a similarity query, not a grep.
As a search problem: given the traffic, find a small set that spans it — and given one failure, find everything that resembles it.
Same machinery you already built for the product, pointed inward. More on the surrounding practice in Measure What Counts.
The pattern
| What it's called | What it actually is | The corpus |
|---|---|---|
| RAG | find the passages that answer this | your documents |
| Tool selection | find the capabilities that match this intent | your tool schemas |
| Agent memory | find what's worth remembering right now | what the agent wrote down |
| Intent classification | find the nearest labeled example | your labeled set |
| Eval curation | find a spanning sample, and the neighbors of a failure | your production logs |
Five names, one operation. Which means one skill improves all five at once — and one blind spot breaks all five at once.
What you can actually do about it
The reframe is only worth anything if it changes what you build. Four moves, roughly in order of how much work they are.
1. Let an LLM pick. Hand the model the candidates and ask it to choose. Expensive, slow, falls apart past a few dozen items — and the right way to start almost every time. It's a working baseline in an afternoon, it tells you whether the task is even well-defined, and every decision it makes is a labeled example you can use to build and evaluate the cheap version later. Teams that skip this step spend a month tuning a retrieval pipeline for a task nobody confirmed was achievable.
2. Move to semantic and hybrid search. Once the baseline works and the bill shows up, replace the wide scan with real retrieval. Not vectors alone: lexical matching still wins outright on identifiers, part numbers, names and error strings, and vectors still win on paraphrase and on the user who describes a symptom instead of naming it. Running both and fusing the results is the 2026 default for good reason — the mechanics, and the specific ways each side fails on its own, are in Hybrid Search.
3. Rerank. Retrieval and ranking are two different jobs and it helps to stop conflating them. The retriever's job is to not lose the answer — cheap, approximate, returns more than you need. Then a reranker reads the query and each candidate together and puts them in order. Retrieve a hundred, keep the ten that earn their place in the window. Your ceiling is set by the retriever's recall and your quality by the reranker's precision, and measuring them separately is how you know which one is actually failing.
4. Make search a tool the agent can call more than once. The classic shape — one query, top-k, stuff the prompt, generate — is a leftover from 2023 context limits that mostly stuck around out of habit. An agent can search, read what came back, notice it asked the wrong way, and search again with better words. Which is what a person does, and it beats one-shot retrieval on anything ambiguous.
This is also where you meet a ranker you don't own. Your agent calls gmail.search, or Zendesk ticket search, or a Drive connector, or some MCP server — and inside each one is a search engine, tuned by another company for another purpose, years ago. Most of them are lexical only. So the agent asks for "angry customers about billing", gets nothing back because no ticket contains the word angry, and reports that it found no issues. Confidently.
You can't fix a ranker you don't own. You can ask it for more results than you need, rerank them yourself, and stop treating an empty response as evidence of absence.
What this looks like on a real product: we built the natural-language layer over Vulcan Cyber's ExposureOS, where a security admin asks for "critical vulnerabilities on Windows servers from last week" and something has to turn that into a query the platform can actually answer. Same problem, wearing a chat interface.
Conclusion
None of this is new. Ranking, relevance, evaluation, and the permanent gap between what someone typed and what they meant — search people have been at this since the nineties, and almost all of it transfers directly.
What's new is where it's hiding. It used to be behind a search box, where at least everyone agreed it was search and somebody owned the metric. Now it's behind a tool call, a memory lookup, a router, a connector — parts of the system nobody labeled as search, so nobody measured them like search either.
Search is an art, and a large share of whether your agent works comes down to how well you practice it. Not the prompt. Not the model. Whether the right thing was in front of it.
Further reading: Lost in the Middle · Context Rot · NoLiMa · Chroma on chunking strategies