Why agent-first · unlob Docs

A search API built for a person is the wrong shape for an agent

A person handed twenty results does something an agent cannot: glances at the domains, recognises two of them, notices that four are the same article, and opens one. That triage is free for them and expensive for you — every discarded result was tokens, and every duplicate was tokens spent twice on the same sentence.

So unlob does the triage on its side of the wire. Three consequences follow, and they are the whole design.

Passages, not pages

A hit is a passage — a contiguous span of a document — and the snippet is that passage’s text, not an ellipsised preview of it. Most questions are answered by the hit itself, with no second fetch.

get_document exists for when you genuinely want the whole page. It is a separate call on purpose: returning full bodies by default would put a hundred kilobytes into your context to answer a question that needed two sentences.

Admission, not ranking

The index does not hold everything and then rank it. It holds what was judged worth keeping, and searching it returns a short list.

This changes how you read an empty result. On an exhaustive index, nothing found means your query was wrong. Here it usually means the index holds little on the topic — which is itself a finding, and why_not will tell you which of the two it is:

curl -sG -H "x-api-key: $UNLOB_API_KEY" https://api.unlob.com/why_not \
  --data-urlencode 'url=https://example.com/the-page-you-expected'

present means it is there and your query missed it. removed names the reason it went. unknown means we never had it.

The cost of this choice, stated plainly: unlob is not the index to use when you need exhaustive recall over the whole web — every obscure forum post, every mirror, every parked domain. It is the index to use when a bounded, high-signal answer is worth more than completeness. If your question is “does any page anywhere contain this string”, you want a different tool.

The signals come with the result

Every hit already carries what you would otherwise compute yourself:

Field	What it saves you
`independent_sources`	Counting distinct hosts asserting a story — the echo-versus-fact test
`host_rank`, `quality`	Deciding whether a source is worth reading before reading it
`centrality`, `in_degree`	Knowing whether this is a central document in its field or a footnote
`story_id`, `group_size`	Collapsing wire copy and mirrors instead of reading them five times
`topics`, `community_id`	Placing a result in the corpus without a second query

They are filterable and sortable, so min_independent_sources=3 is a corroboration filter that costs nothing extra, and sort=centrality is trust-triage in one parameter.

The graph is the part that replaces a loop

Search finds a passage. The Coverage Graph tells you what it is connected to: who else reports it, what entity it concerns, what sits between it and something else, and — with assemble_context — a whole packed, deduplicated, trust-ranked reading set for a question.

That last one is the honest summary of the product. An agent that would otherwise run search → dedupe → corroborate → rank → pack itself can make one call and read the result.

What this means for your prompt

Two habits are worth building in:

Check partial. When it is true, part of the corpus was unreachable for that request. A short answer is then a symptom, not a finding. This is the one field that changes what a result means.
Corroborate before asserting. One rumour reprinted forty times and one finding reported by four newsrooms look identical in a result list. They do not look identical to corroborate.