Coverage transparency

Why this exists

The index is admission-controlled: it holds what was judged worth keeping rather than everything that exists. That makes an empty result ambiguous in a way it would not be on an exhaustive index. Three quite different situations look identical from the outside:

• The page is in the index and your query missed it.
• The page was in the index and was dropped.
• The page was never there.

why_not tells you which.

curl -sG -H "x-api-key: $UNLOB_API_KEY" https://api.unlob.com/why_not \
  --data-urlencode 'url=https://docs.rs/tokio'

The three answers

Present — it is there. Your query or your filters missed it; the hit comes back so you can see what it looks like.

{ "status": "present", "hit": { "id": "…", "url": "…", "snippet": "…" } }

Removed — it was there and went, with the reason and, where one exists, the passage that replaced it.

{
  "status": "removed",
  "reason": "superseded",
  "superseded_by": "…:0",
  "removed_at": 1735689600
}

Unknown — never admitted, or never seen.

{ "status": "unknown" }

The removal reasons

reason	What it means for you
redundant	Something else in the index already covered this. Search for the topic; you will find the survivor.
superseded	A newer or better version replaced it. superseded_by names it.
stale	It aged out.
access-starved	It stopped being reachable — the source went away, or stopped serving it.
lost-replacement	The thing that superseded it was itself removed.
tombstoned-source	The source is excluded.

The first two are the common ones, and neither means the information is gone. redundant and superseded both point at something still in the index that covers the same ground.

Using it

When a search comes back empty, ask about a URL you expected to be there. present means loosen your filters. unknown means the index genuinely does not hold this, and you should say so rather than inferring the web is silent.

Before asserting absence. “There is no public documentation of X” is a strong claim. unknown on the pages you would expect makes it defensible; an empty search result on its own does not.

When a result you used before disappears. removed with superseded_by hands you the replacement directly, which is usually what you wanted anyway.

Also check partial on the search that came back empty. If it was true, part of the corpus was unreachable and the emptiness is a transport problem, not a coverage one — see Reading a result.

What it does not tell you

why_not reports what happened to a URL. It does not report the criteria by which admission decides — that is not a caller-observable property, and it is not documented anywhere on this site. What you get is the outcome and the reason class, which is what you need to act.

Next

• Reading a result — partial, and the rest of the response.
• Search recipes — the “when the answer is empty” section.
• why_not reference