Census / live & auto-updating
The agent-facing web — a census
The rest of this site measures the agents that visit us. This page turns that around: FomiteBot, a small, disclosed, honestly-identified crawler, visits sites that already reached out to agents — their own +https:// calling card, left in their User-Agent, is how we found them — and records, structurally, which agent-facing surfaces they actually expose. See /methodology for exactly how a probe is scored.
1. Which surfaces sites actually expose
“Present” means a probe of that exact path returned a structural presence verdict — not a 200. A homepage re-served at every path (a soft-404) is scoredabsent, not present; a robots.txt Disallow is scored disallowed, never routed around.
| Surface | Sites exposing it | % of crawled |
|---|---|---|
| llms.txt | 4 | 26.7% |
| ai.txt | 0 | 0.0% |
| .well-known/mcp.json | 0 | 0.0% |
| .well-known/ai-plugin.json | 1 | 6.7% |
| .well-known/agent.json | 0 | 0.0% |
| /mcp | 1 | 6.7% |
| security.txt | 2 | 13.3% |
| openapi.json | 0 | 0.0% |
2. How robots.txt treats us
fail-closed means the site's own robots.txt could not be fetched or parsed cleanly and we treated that as a refusal rather than assume permission; an allow-all after a 404 means no robots.txt existed at all.
| robots.txt outcome | Sites |
|---|---|
| Parsed normally | 10 |
| No robots.txt (allow all) | 0 |
| No applicable group | 0 |
| Fail-closed (fetch/parse failed) | 5 |
| Other | 0 |
3. Site by site
Every crawled site, most recently crawled first, and what it exposes.
| Host | Source | Blocks DC? | llms.txt | ai.txt | .well-known/mcp.json | .well-known/ai-plugin.json | .well-known/agent.json | /mcp | security.txt | openapi.json |
|---|---|---|---|---|---|---|---|---|---|---|
| netapi.com | seed | — | yes | — | — | — | — | — | — | — |
| aive.global | seed | — | — | — | — | — | — | — | — | — |
| mcpbeat.com | seed | blocks | — | — | — | — | — | — | — | — |
| support.anthropic.com | seed | — | — | — | — | — | — | — | — | — |
| www.baidu.com | seed | — | — | — | — | — | — | — | — | — |
| openai.com | seed | blocks | — | — | — | — | — | — | — | — |
| api.slack.com | seed | — | yes | — | — | yes | — | — | yes | — |
| awario.com | seed | — | — | — | — | — | — | — | — | — |
| code.google.com | seed | — | — | — | — | — | — | — | — | — |
| roninforge.org | seed | — | yes | — | — | — | — | — | — | — |
| sec.sqrx.io | seed | — | — | — | — | — | — | — | — | — |
| www.domainarrivals.com | seed | — | — | — | — | — | — | — | — | — |
| github.com | seed | — | yes | — | — | — | — | yes | yes | — |
| immersivecommons.com | seed | — | — | — | — | — | — | — | — | — |
| smithery.ai | seed | — | — | — | — | — | — | — | — | — |
Cite this
Every figure here is recomputed live and is CC0. Take the JSON at /api/agent-web, read how it is measured (with the weaknesses stated) at /methodology, and read about the crawler itself, how to verify it, and how to block it at /fomitebot. If your site is listed here and something is wrong, tell us and we will fix it and say that we did.
The Fomite is a field station on the agentic web. Start here · FomiteBot, the crawler · the Observatory.