Methodology and evidence policy

Evidence labels

Three directory evidence states are independent: a listing can hold any combination. Outside results get their own types. A stale flag marks an observation whose version or endpoint has since changed; it is kept for history and no longer counts.

Publisher documented

A first-party source explains the capability. Nothing was run. Registry listings and package metadata stop here.

Connection checked

A named client reached an allowed endpoint over a named protocol on a recorded date, and saw the expected capability. It says nothing about task results.

Task passed

A named fixture ran, and an independent read of the downstream result matched what was expected. Failed and incomplete are recorded the same way.

Third-party benchmark

An outside publisher’s result, shown with its setup, sample, date and denominators. Not run by AgentsByTask.

Vendor-reported

A vendor’s own claim about its own product, with the vendor’s stated sample.

Stale

The version or endpoint changed after this observation. It is kept for history and no longer counts.

What each label requires

• Publisher documented: a first-party source identifies the action and any known prerequisites. We record only what the source states, with its link and the date we read it.
• Connection checked: a named, authorized client observed the protocol connection and the relevant capability at a dated endpoint. For MCP that means initialize and tools/list. An OAuth challenge proves authentication is required, not that an authenticated tool list works.
• Task passed, failed or incomplete: an authorized fixture ran and an independent read of the destination confirmed (or did not confirm) the expected artifact, including any required human approval. We record the fixture and tool versions, environment, date, case counts, verifier and approval point. A success message from the tool alone is never enough.
• Third-party benchmark: a named outside source published a result with an inspectable method. We show its publisher, date, setup, sample and denominators, and keep it apart from directory runs.
• Vendor-reported: a vendor’s own result about its own product, with its stated sample.

Discovery can only ever create publisher-documented records. The database enforces this: a record whose origin is discovery cannot hold any other evidence type, and evidence rows are append-only apart from the stale flag.

Ranking and cost

• Only directory task runs are ranked, and only within the same fixture and environment, with outcome first. Listings without comparable runs are shown in labeled groups in research order, which is not a ranking.
• Outside benchmarks are shown in their publisher’s terms. Where we sort them, the caption says by which published measure.
• Deprecated listings are excluded from rankings. Deleted listings are removed from public pages; their past evidence is preserved.
• Published prices keep the publisher’s unit (per minute, per call, per month, per page) and the date we checked them. They are never converted into a cost per completed task.
• Cost per completion is shown only when a comparable run recorded actual metered spend, including failed attempts and retries, and a verified completion count. Zero completions or missing spend means no figure.

Continuous discovery

A scheduled Cloudflare Worker reads the official MCP Registry and a short list of approved publisher sources (Agent Skill metadata, release metadata, OpenAPI documents and published WebMCP demos). It is bounded per run, resumable and idempotent:

• Only HTTPS URLs on an exact allowlist are fetched, with timeouts, byte caps, content-type checks and redirects re-validated against the same allowlist. No visitor or registry record can supply a URL to fetch.
• Registry sync follows nextCursor pages within a window, saves progress after each page, and advances its watermark only when a window completes. Records are keyed by namespace and version; older updates never overwrite newer ones. Deprecations and deletions come only from the Registry’s explicit status, never from a missing row.
• Publisher descriptions are untrusted data. They are length-capped, escaped and quoted, never followed as instructions. Nothing discovered is installed, executed or called.
• No model or paid API is used. Classification is deterministic and written down below.

How tasks are created

Discovery matches documented actions to existing tasks and can propose a new task or category when a source describes a distinct outcome. Each candidate keeps the source, observation date and reason it belongs. A new page stays pending until at least 2 independent publishers document the same outcome with enough detail to compare the actions. When a generated page loses that support, it returns to pending and leaves the sitemap. Individually approved publisher sources can have their own listings.

Tests we run, and tests we never run

Directory fixtures run locally or against an authorized test tenant. We never call production writes or sign-in-only actions to test a listing, never sign in to a third-party account, and never send a real transaction. The reference PDF-to-CRM fixture is an author-run local fixture, labeled as such; it is not a vendor task pass.

Owned products

Soxoa LLC operates AgentsByTask and some listed products. Those listings carry a disclosure and meet the same evidence requirements as every other listing.