Crawl-Only Onboarding — What to Send Us

A one-page checklist for a client onboarding via the crawl-only route — you share nothing (no code, no repo, no backend, no secrets). We read your public website exactly as any visitor would and build an agent-readable surface from it. If you later want agents to actually do things against your systems (submit forms, place orders), that's the full-access route — ask us for that checklist.

What you get: a read-only agent surface at agents.<your-domain> — discovery (llms.txt, manifest), machine-readable docs/context, a changelog, and read actions (get business info, list pages, search content) derived from your public content. What you do not get on this route: typed actions that change data in your backend (those need the full-access route). We don't rebuild or modify your website; the only thing we ask you to add is three one-line edits so agents and AI search can find the surface (see below).

What we need from you (short list — that's the point)

• Your public website URL (the homepage we start from, e.g. https://acme.com).
• Permission to crawl it, and any paths you'd like us to avoid (e.g. /admin, staging areas, anything behind login).
• The agent subdomain you want — typically agents.<your-domain> — so you can point a CNAME at us. That one DNS record is the only change we require on your side.
• A technical contact who can add that one DNS record and confirm go-live.
• (Optional) Business name + one-line description. We can pull these from your site, but sending them avoids guesswork.
• (Optional) A rough page budget if your site is large (we crawl up to a set number of pages; tell us the sections that matter most).

That's the whole requirement for us to build and serve your surface. The next section is what makes it findable — three lines on your own site, and the biggest single difference between a surface that exists and one that gets used. Everything after that just makes the result better.

Three one-line additions so agents can actually find you (strongly recommended)

Your agent surface ships every discovery file an agent looks for — llms.txt, a manifest under /.well-known/, HTTP Link: headers — and they all fire the moment something visits agents.<your-domain>. Getting visited in the first place is the half that lives on your side, and it takes three lines. Search engines learn about a new address by following ordinary links and reading sitemaps: give them one of each, and your agent surface enters the index that ChatGPT, Claude and Gemini answer from.

Three additive one-liners, a few minutes of your web person's time, no code:

• One link from your website to your agent surface — an ordinary <a href="https://agents.<your-domain>/"> anchor, with text like "For AI agents" or "Agent & API access". A footer link is enough: it does not need to be in your main navigation, and most human visitors will never notice it. This single link is the one mechanism every search crawler on earth follows.
• One line in your robots.txt: Sitemap: https://agents.<your-domain>/sitemap.xml. Unlike a comment, a Sitemap: directive is a real machine-read instruction — and it is the only discovery mechanism robots.txt actually has.
• One entry in your sitemap.xml for https://agents.<your-domain>/.

We hand you the exact text. Once your surface is live we publish a ready-to-paste pack at https://agents.<your-domain>/docs/discovery-pack.md: all three snippets pre-filled with your real domain, the search-console steps below, and your MCP Registry namespace. Your web person copies and pastes rather than composing anything.

One step only you can take: submit https://agents.<your-domain>/sitemap.xml in Google Search Console and Bing Webmaster Tools. Verification is restricted to the domain owner, so we can't do it for you. Do both — Bing feeds ChatGPT's search, Google feeds Gemini.

What you lose without them

Nothing breaks. Your surface still works perfectly for any agent handed the URL directly, and for anything that finds you through the MCP registry. What won't happen is your surface appearing in Google or Bing — and because ChatGPT, Claude and Gemini answer questions like "does this company have an agent API?" by querying a conventional search index rather than crawling the live web, it won't come up there either. This is not a limitation of our platform; it's how search discovery works for any new address on the web.

What's still true: we need no code access, no repo, no backend and no secrets for any of this. The three edits are purely additive, and we never touch your source, servers or databases.

Business details for AI search (optional — recommended)

We automatically publish a machine-readable entity record of your business (schema.org structured data) that AI search engines — ChatGPT, Perplexity, Google's AI overviews — use to identify and quote you accurately. From your website alone we can derive the basics; if you send the details below, the record is upgraded to the kind search engines show rich results for:

• Business type (e.g. local business, clinic, professional service).
• Registered/legal name (if different from your brand name).
• Phone number and full address (street, postal code, country).
• Opening hours (if you have physical premises or fixed service hours).
• Your official profiles — LinkedIn, Facebook, Instagram, Google Business Profile URLs (these disambiguate you from similarly named businesses).
• Your key services/offerings — name + one line each (5–10 is plenty).

None of this is published anywhere it isn't already public — it just makes the machine-readable version authoritative instead of guessed. We'll also send you a small snippet your web person can paste into your site's <head> so classic search engines pick it up too — one paste, alongside the three additions above.

Make your crawl produce a great surface (optional, but it raises your audit score)

The higher-quality your public HTML, the richer and more accurate your agent surface. Ideally your live site has:

• A sitemap.xml (we look for it at https://<your-domain>/sitemap.xml) — this is how we find all your important pages instead of just what's linked from the homepage.
• Server-rendered HTML — real content in the page source. If your site is a client-rendered single-page app (content only appears after JavaScript runs), our crawler sees mostly empty pages. Server-side rendering or prerendering fixes this.
• A <title> and <meta name="description"> on each page.
• Clear <h1>/<h2> headings and readable body text (this is what agents read).
• Internal links between pages (we follow same-site links to discover content).

What we read vs. never touch

• We read: your public pages' <title>, meta description, <h1>/<h2> headings, visible text, and same-site links — the same things a browser or search engine sees.
• We never touch: your source code, servers, databases, admin areas, anything behind a login, or any secrets. On this route you literally hand us nothing but a URL.

How freshness works

Your agent surface can go stale if your site changes and we don't re-read it. On the crawl-only route we keep it current by re-crawling on a schedule. Tell us:

• How often your content meaningfully changes (so we can set the re-crawl cadence).

Want agents to do things, not just read?

Crawl-only gives agents a read-only view. If you want them to submit enquiries, create orders, or call your services, you'll want the full-access route (share a static export or an OpenAPI spec, plus a backend endpoint we can call). Ask us for the Full-Access — What to Send Us checklist.