How to Generate a Retrieval Eval Set From Your Own Corpus

The problem with the eval you just wrote

So you built your semantic search. You write fifteen test queries. You run them, eyeball the results, and they look good.

Of course they look good. You wrote the queries after seeing what the system does. Every hand-written eval is contaminated by the author's model of the system, and it measures the queries you happened to imagine rather than the ones your users will type.

There is a second, quieter problem. A hand-written suite usually asserts something weak — "did we get a result", "does the top hit look related". It almost never asserts which document should have come first, because establishing that by hand is hard.

Just try asking a customer or a stake holder to generate a ground truth set and 9 out of 10 times, after a week or so, you will receive an LLM generated list of questions you could have built yourself. It a hard question to ask and a harder question to answer.

In any case, the suite cannot detect the failures that actually matter: consistently returning plausible-but-wrong results.

We hit this problem any time we build a searching agent that uses semantic or hybrid search — the same corpus we later used to measure domain filtering and reranking. This specific post uses a search over patent claims system as an example, but this is something we stumble upon at least once a month.

The trick

The trick is generating at least a portion of the evaluation corpus yourself. I know it might sound odd - I mean isn't it kind of circular - but follow me here.

1. Take a document that is definitely in your index.
2. Ask a model to describe it the way a normal person would, in their own words. Then check whether search finds that document back.
3. The document is the ground truth. You did not choose it, so you cannot bias toward what works. And it gives you precision@k against a known answer rather than a vague "looks relevant".

precision@k, in one paragraph. Of the top k results a search returns, what fraction are relevant:

$$\text{precision@}k = \frac{|\,\text{relevant} \cap \text{top-}k\,|}{k}$$

In this setup each generated description has exactly one correct answer — the document it was written from — so the number degenerates into something simpler and more useful: did the source document make it into the top k, on what share of cases. That is why $\text{precision@}3$ here can read 66.7% rather than being capped at 1/3. $\text{recall@}10$ is the same question at depth 10, and MRR (mean reciprocal rank) averages $1 / \text{rank}_{\text{source}}$ across cases, so it rewards being first rather than merely present.

For patents, the prompt looks roughly like this:

Below is a patent. Write how the INVENTOR might have described this idea in
plain English, before they ever spoke to a patent attorney.

Rules:
- Two or three sentences. Everyday language.
- Describe what it does and roughly how, not what it is called.
- Do NOT reuse the patent's distinctive technical terms. If the patent says
  "electrically powered signal emitter", say "a light that comes on".

A real generated case:

Patent: Bidirectional amplifier including matching circuits having symmetrical...

Generated description: "I built an amplifier that can boost a signal traveling in either direction — out or back in — using the same circuit, instead of needing separate paths for each direction."

That is a fair test. Nobody would type the patent's title into a search box.

The filter that makes it honest

Here is the part that is easy to skip and ruins everything if you do.

Models paraphrase lazily. Ask for a description and you will often get the source text lightly reworded, keeping all its distinctive vocabulary. If the description shares rare words with the document, BM25 alone will ace your eval and you will learn nothing about semantic retrieval. You will have built an expensive keyword-matching test.

Leak, and how we measure it. A leak is distinctive vocabulary the generated description copied straight out of the source document. Take the rare words of each — lowercase tokens of five letters or more, minus a common word list — and ask how much of the description's rare vocabulary the source already contained:

$$\text{leak} = \frac{|\,\text{rare}(\text{description}) \cap \text{rare}(\text{source})\,|} {|\,\text{rare}(\text{description})\,|}$$

0 means the description shares no unusual words with the document and a keyword engine has nothing to lock onto; 1 means every distinctive word was lifted verbatim, and the case tests string matching rather than meaning.

So measure the leak and reject the case if it is too high:

def leak_ratio(description: str, claim: str) -> float:
    """How much rare vocabulary the description copied from the source."""
    def rare(text):
        words = re.findall(r"[a-z]{5,}", text.lower())
        return {w for w in words if w not in COMMON}

    d, c = rare(description), rare(claim)
    return len(d & c) / max(len(d), 1)

Anything over about 0.35 gets thrown away and regenerated. In our run, mean leak was 0.14 and roughly one case in forty was rejected.

Without this filter the eval is theatre. With it, a passing score means the semantic layer is genuinely doing work.

What you get

Real metrics, on real documents, with no hand-labelling:

ground truth: 39 cases, depth 10
  precision@1  53.8%
  precision@3  66.7%
  recall@10    82.1%
  MRR          0.620

Three things fall out of this that a hand-written suite would not have given us.

Coverage you did not choose. Because the cases come from randomly sampled documents, the mix of subjects matches the mix in the index rather than the mix in our heads. Our 39 patents landed as 16 electrical, 12 computing and physics, 6 mechanical — which is roughly what US patents actually look like. Left to ourselves we would have written test cases about kitchen gadgets and drones, because those are the ones easy to imagine, and we would have tested none of the dense electrical and software patents, exactly the ones whose language looks least like how a person would describe them.

A straight answer to "is it broken, or is it just missing?" When a search returns nothing useful, there are two very different explanations: the search is bad, or the document simply is not in the index yet. Guessing between them wastes days. We were still loading patents in, and with 4% of them indexed a "smart mousetrap" query came back empty. At 10% the same query returned the right patent first. Nothing about the search had changed — it was only ever a coverage problem, and we stopped debugging a system that worked.

Numbers you can publish. Most search products claim to be good. Very few publish precision@1, because it is usually lower than the marketing implies.

What it does not tell you

Be honest about the limits.

The descriptions are model-written, so they inherit that model's idea of how a layman writes. Real users type shorter, vaguer, worse-spelled queries. This is a good relative measure for catching regressions, not an absolute prediction of production quality.

It also only tests retrieval. If your product has a reasoning layer on top — and if you are doing anything beyond ranking, it should — that needs its own eval, because the two fail in completely different ways.

Doing it yourself

The recipe generalises to any corpus where documents have a natural description:

1. Sample documents at random from your index
2. Have a model describe each in the user's voice, forbidding source vocabulary
3. Reject any description that leaks too much rare vocabulary
4. Search with the description, record the rank of the source document
5. Report precision@1, precision@3, recall@k, MRR

An hour of work, a few cents of inference, and you can finally answer "did that change help?" with a number instead of an opinion.