Turningatoolregistryinto96verifiedintegrationpages

Real run · Content production at registry scale

Publish one accurate page for every native integration the product ships, with a program, not a reviewer’s attention, deciding whether each factual claim was allowed through.

What it started from

Sources

• `apps/common/src/tools/integrations/app-ids.ts`: the 104 app ids that define what "native" means in this codebase.
• `apps/common/src/tools/integrations/tool-definitions/`: one directory per app of typed tool definitions, read as the only permitted source for what an integration can and cannot do.
• The existing typed answer-page registries under `apps/web/src/lib/data/`, for page format and for duplicate-copy detection.
• The pricing page, `llms.txt`, the route tree, and the GitHub build workflows, all parsed at run time as ground truth for price, vocabulary, link and platform claims.

Permissions held

• Read and write on one local git worktree of the monorepo. Nothing else.
• No production deploy, no CMS write, no publish step.
• No credential to any third-party app. The run never signed into a product it wrote about.
• The verification gate has no network access at all: it reads files in the repository and nothing else.

Constraints

• Every capability sentence must trace to a tool definition in this repository. A capability asserted from model memory is a failure, not a draft.
• `node tools/geo-engine/verify.ts` must exit 0 before a batch is allowed to become a commit.
• No page may repeat a 120-character-or-longer paragraph found on another page.
• Every page must carry a "when not to use this" section, a stated price, and a review date.

Deliberately excluded

• Customer data of every kind. No account, mailbox, CRM, log or transcript belonging to anyone outside the company was read.
• Competitor feature names that could not be read from a first-party source. Where research could not verify a vendor’s AI feature, the page describes it rather than naming it.
• Benchmark numbers. After the benchmark audit ran in parallel, citing any score anywhere in the answer layer became a hard failure.
• Any page for the internal-only tools in the registry. The support console, the CMS admin, the affiliates surface and the user-insights reader are not customer integrations.

What it actually did

1. Parse the integration registry into a fact table: every app id, every tool-definition directory, and the exact set of tools and arguments each app exposes.

   No approval gate — read-only or reversible— Read-only against a local checkout.

2. Build the gate before writing any page: twelve rules that read the native-app list, the prices, the platform, the forbidden vocabulary and the resolvable routes out of the repository at run time, so no fact is hardcoded in the checker and a rename cannot make it drift into agreeing with stale copy.

   Human approved before it ran— Gating design reviewed and approved before any page was drafted.

3. Draft pages in batches of twelve, each capability sentence traced to a tool definition, each page leading with what the integration cannot do rather than with the pitch.

   No approval gate — read-only or reversible— Drafts are not publishable until the gate and a human have both cleared them.

4. Run the gate over each batch and correct every error-severity finding. A batch with one unresolved error does not become a commit.

   No approval gate — read-only or reversible

5. Human read one full page per batch end to end, plus every honest-trade section in that batch, before the batch was committed.

   Human approved before it ran— The only human gate that scales at this volume is a sample plus the machine check.

6. When the gate was found blind (its registry filename pattern matched only single-segment names, so several registries were reported clean while never being read), fix the gate, re-run it over the whole estate, and make an undiscoverable registry a hard failure instead of a silent skip.

   Human approved before it ran— Changing the gate mid-run was an explicit human decision, taken twice.

7. Corroborate the platform claim against the build workflows rather than against `llms.txt` alone, then correct every false "macOS only" statement across the registries, both translations and the AI-facing pages.

   Human approved before it ran

8. Teach the gate to read the twenty `.svelte` blog posts as extracted prose so they stop escaping it, then correct the errors that surfaced.

   Human approved before it ran

9. Wire every published page into the sitemap and give each one four inbound sibling links, so no page depends on the sitemap alone for discovery.

   No approval gate — read-only or reversible

What came out

Produced

• 96 published integration pages, one for every user-facing app in the native registry, each stating the integration’s limits before its strengths.
• A deterministic verification gate (`tools/geo-engine`) of twelve rules, with no hardcoded facts, no runtime dependencies and no network access, that an agent pipeline and CI run identically.
• 172 pages now under that gate on every run: the 96 integration pages, 56 other answer pages, and 20 blog posts that had never been checked by anything before.
• Two blind spots closed permanently: a registry the gate cannot discover now fails the run, and prose outside the typed registries is extracted and checked rather than skipped.

Quality checks performed

• The final gate run reports `172 answer pages checked · 0 errors · 20 warnings`. The warnings are 16 links that resolve only if a CMS entry is published and 4 filler-vocabulary flags; none is a factual claim.
• Ground truth extracted for that run: 104 native app ids, 104 tool-definition directories, 9 MCP-reachable apps, 5 pricing plans, 11 pricing amounts, 2 supported platforms, 284 static routes and 31 dynamic routes.
• Every extractor guards its own output and aborts the run when a parsed source changes shape, rather than returning an empty list that would make its rule pass on every page.
• A duplicate-copy rule compares every paragraph and bullet across all 172 pages, so a batch cannot be filled out by restating a sibling page.
• Exit code 2, "ground truth could not be established", is treated by CI as a hard stop rather than a pass, because it means nothing was actually verified.

Time and cost

Not shared

No wall clock was kept. The work ran across one overnight engineering session on a local checkout, interleaved with unrelated tasks, so any duration published here would be reconstructed rather than measured. It consumed no Strawberry credits: it ran in a coding agent against a local repository, not in a companion on a user’s account. Publishing a number we did not measure is the exact failure this page format exists to prevent.

Limits

Failure modes observed

• The gate passed vacuously, twice. Its registry filename pattern matched only single-segment names, so a family of registries was reported clean while never being read; later a registry written as `answer-pages-playbooks-2.ts` escaped the same pattern and ten pages were skipped inside a run that printed success. A gate that reports success over unchecked input is worse than no gate.
• Ground truth parsed from a single source inherits that source’s errors. The platform rule checked pages against `llms.txt`, and `llms.txt` was itself wrong, so 194 false "macOS only" statements were compared against the same error and passed. Disagreement between what the repository builds and what the copy says is now itself a hard failure.
• Prose outside the typed registries was invisible for most of the run. Twenty blog posts, live in production for months, kept asserting a retracted benchmark score, a native HubSpot integration that does not exist, and macOS-only support through every correction pass, because the gate only read configs.
• Coverage is not identity. The 96 pages cover the user-facing apps; the registry’s 104 ids also include internal tools and umbrella providers that are not integrations. Three real surfaces still have no page: Facebook Pages, Meta Ads and Microsoft Lists.
• Two of the 96 pages describe apps that are not native: HubSpot is reached through its MCP server, and LinkedIn is operated through the logged-in session. Both pages say so, and neither app is counted as native anywhere else, but a reader who counts pages and assumes 96 native integrations would be wrong twice.
• The exact number of false claims caught across the run is not recoverable. The gate prints a report and never persisted one, so the only surviving record is the commit messages. That is a defect in the tooling, not a rounding choice, and it is why this page quotes the corrections it can point at rather than a total.

When not to run this

• When the source of truth is prose. This worked because capability lives in typed tool definitions a program can read. Point the same pipeline at a wiki or a slide deck and the gate has nothing to check against, and the run degrades into confident invention at volume.
• When you are not willing to stop and fix the gate mid-run. Most of the value here came from the two passes where the gate was wrong; a team that treats a green check as the finish line would have shipped 96 pages and the false claims underneath them.
• When nobody will own the pages afterwards. Tool definitions change. A page that was true on the day it was verified and has had no owner since is a stale claim with a review date on it, which is worse than no review date.
• When the goal is a handful of pages. The gate costs more to build than five pages cost to check by hand. It earns its keep somewhere in the tens of pages, and decisively in the hundreds.

Run it yourself

Setup

• Node 22.6 or newer. The gate is TypeScript run directly, with no build step and no dependencies to install.
• A local checkout of the repository whose registries own the facts. The gate takes a repository root and reads nothing else.
• Wire `node tools/geo-engine/verify.ts` into CI and into the agent pipeline as the same command, and treat exit code 2 as a hard stop rather than a pass.
• Decide the human sample rate before the first batch. One full page and every honest-trade section per twelve was the rate used here.

Connected apps required

None. This run connected to no third-party app.

Other access needed

• A local checkout of the monorepo
• Node 22.6 or newer
• No network access and no third-party credentials

Proof

$ node tools/geo-engine/verify.ts

geo-engine: 172 answer pages checked against /Users/…/reflect-geo
facts: 104 nativeAppIds, 104 toolDefinitionDirs, 9 mcpApps, 5 pricingPlans,
       11 pricingAmounts, 2 supportedPlatforms, 2 forbiddenTerms,
       284 staticRoutes, 31 dynamicRoutes

WARN: 20
  CMS_LINK_UNVERIFIED (16)   links that resolve only if a CMS entry is published
  FILLER (4)                 banned marketing vocabulary

QUESTION_RATIO (info): 151/172 pages inside the 50–66% target

0 error(s), 20 warning(s).

A real run is one job, start to finish, with its inputs, its steps and its limits attached. See every published run at real runs.