Why agent-first search beats ten blue links

A search API built for a person is the wrong shape for an agent

A person handed twenty results does something an agent cannot: glances at the domains, recognises two of them, notices that four are the same article, and opens one. That triage is free for them and expensive for you — every discarded result was tokens, and every duplicate was tokens spent twice on the same sentence.

So unlob does the triage on its side of the wire — and then goes one step further: it says what the evidence is, and whether it is enough. Four consequences follow, and they are the whole design.

Passages, not pages

A hit is a passage — a contiguous span of a document — and the snippet is that passage’s text, not an ellipsised preview of it. Most questions are answered by the hit itself, with no second fetch.

get_document exists for when you genuinely want the whole page. It is a separate call on purpose: returning full bodies by default would put a hundred kilobytes into your context to answer a question that needed two sentences.

Admission, not ranking

The index does not hold everything and then rank it. It holds what was judged worth keeping, and searching it returns a short list.

This changes how you read an empty result. On an exhaustive index, nothing found means your query was wrong. Here it usually means the index holds little on the topic — which is itself a finding, and why_not will tell you which of the two it is:

curl -sG -H "x-api-key: $UNLOB_API_KEY" https://api.unlob.com/why_not \
  --data-urlencode 'url=https://example.com/the-page-you-expected'

present means it is there and your query missed it. removed names the reason it went. unknown means we never had it.

The cost of this choice, stated plainly: unlob is not the index to use when you need exhaustive recall over the whole web — every obscure forum post, every mirror, every parked domain. It is the index to use when a bounded, high-signal answer is worth more than completeness. If your question is “does any page anywhere contain this string”, you want a different tool.

The signals come with the result

Every hit already carries what you would otherwise compute yourself:

Field	What it saves you
independent_sources	Counting distinct hosts carrying a story — a cheap syndication signal; ground computes real independence
host_rank, quality	Deciding whether a source is worth reading before reading it
centrality, in_degree	Knowing whether this is a central document in its field or a footnote
story_id, group_size	Collapsing wire copy and mirrors instead of reading them five times
topics, community_id	Placing a result in the corpus without a second query

They are filterable and sortable, so min_independent_sources=3 is a corroboration filter that costs nothing extra, and sort=centrality is trust-triage in one parameter.

The graph is the part that replaces a loop

Search finds a passage. The Coverage Graph tells you what it is connected to: who else reports it, what entity it concerns, what sits between it and something else, and — with assemble_context — a whole packed, deduplicated, trust-ranked reading set for a question.

Evidence, not links, and a receipt

ground is the honest summary of the product. An agent that would otherwise run search → dedupe → corroborate → rank → pack → judge itself makes one call and reads the result: evidence items that say who originally published them, who republished them, what kind of source they are and whether they are independent of one another, packed to a token budget, with a coverage receipt that says what was searched, what was not, and whether the set is enough. Every disambiguation and triage loop an agent runs costs latency, retrieval calls and context tokens; this moves that work into deterministic retrieval infrastructure, and leaves the reasoning to whichever model you use.

What this means for your prompt

Two habits are worth building in:

• Check partial. When it is true, part of the corpus was unreachable for that request. A short answer is then a symptom, not a finding. This is the one field that changes what a result means.
• Corroborate before asserting. One rumour reprinted forty times and one finding reported by four newsrooms look identical in a result list. They do not look identical to corroborate — and ground goes further, counting origins with distinct owners rather than hosts.

Next

• Grounding — the one call.
• Reading a result — every field on a hit.
• The Coverage Graph — the six calls.
• Search recipes — this, applied.
• Prompting an agent to use unlob well — this design, as instructions