Apify MCP coverage contract

This companion project validates an Actor result after an AI agent calls it through the hosted Apify MCP server. It checks execution, requested-source coverage, and dataset invariants separately.

The example comes from a bounded commercial real estate workflow: Codex asked one Actor for five office listings from Crexi and five from LoopNet in a Dallas sale search. The first platform-successful run returned only the Crexi half. The final run returned both halves. This verifier makes that difference machine-readable instead of treating SUCCEEDED as the whole verdict.

What this repository proves

The verifier reads four local JSON files:

1. examples/job.json defines the requested sources, filters, per-source target, and charge ceiling.
2. run.json records the platform status, MCP origin, runtime configuration, and actual Apify usage cost.
3. run-summary.json records the Actor's source-level outcomes.
4. dataset.json contains the final normalized rows.

It returns three independent checks:

• execution: Did the run succeed through MCP and remain within the declared charge ceiling?
• coverage: Did every requested source succeed and contribute the requested bounded result count?
• data: Do the rows match the requested state, transaction, asset class, and source constraints, with unique listing IDs and URLs?

safeToUse becomes true only when all three checks pass. A platform status of SUCCEEDED can therefore coexist with coverage: partial and safeToUse: false.

The term coverage contract describes the method used in this project. It is not an official Apify or Model Context Protocol standard.

Prerequisites

• Node.js 22 or newer
• npm
• jq only for the optional fixture syntax check
• An MCP client that supports Streamable HTTP and OAuth if you want to connect to Apify yourself

The local verifier never contacts Apify and does not require an Apify account. The MCP connection is needed only if you reproduce the agent-side workflow with your own run and your own budget.

Connect one Actor and its audit tools through OAuth

The portable example in mcp-config.example.json scopes the hosted server to a single Actor plus the four run and storage tools used by the audit. The hosted server automatically adds get-actor-output to Actor-related configurations:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=kazkn/commercial-real-estate-brokerage-intel,get-actor-run,get-actor-log,get-dataset-items,get-key-value-store-record"
    }
  }
}

With Codex CLI, the equivalent commands are:

codex mcp add apify --url "https://mcp.apify.com?tools=kazkn/commercial-real-estate-brokerage-intel,get-actor-run,get-actor-log,get-dataset-items,get-key-value-store-record"
codex mcp login apify

The second command opens the browser-based Apify authorization flow. This keeps credentials out of project files. Restart the client or open a fresh task if it does not refresh its tool list after authentication.

When reproducing the workflow, set a charge ceiling before calling the Actor, keep the requested result count small, and use the explicit audit tools to inspect the stored input, platform metadata, source summary, logs, and dataset. A successful OAuth login proves authentication, not result quality.

Run the local verifier

Install dependencies, run every test and privacy check, then build the CLI:

npm install
npm run verify
npm run build
node dist/src/cli.js examples/job.json fixtures/complete/run.json fixtures/complete/run-summary.json fixtures/complete/dataset.json

To inspect the platform-successful but incomplete example:

node dist/src/cli.js examples/job.json fixtures/partial/run.json fixtures/partial/run-summary.json fixtures/partial/dataset.json

The complete fixture exits with code 0. A valid but unsafe workflow verdict exits with code 2. Invalid JSON or a schema violation exits with code 1, and incorrect CLI usage exits with code 64.

Read the verdict

The complete fixture prints:

{
  "execution": "pass",
  "coverage": "complete",
  "data": "pass",
  "safeToUse": true,
  "reasons": []
}

The partial fixture passes the execution and row checks but reports three coverage reasons: the Actor summary is partial, the LoopNet outcome timed out, and the dataset contains zero of the five requested LoopNet rows. The returned rows can still be a valid Crexi shortlist, but they cannot be described as a complete two-source result.

The checks are deliberately separate. A failed execution is a technical failure. A completed execution with a missing requested source is partial coverage. Rows outside the declared filters or duplicated identities are data failures. Keeping those states separate prevents an agent from turning a source outage into a market conclusion.

Fixture provenance

The counts, statuses, timeouts, and costs mirror a bounded six-run experiment performed through the Apify MCP server on August 5, 2026. Listing identifiers, URLs, locations below state level, and contacts are deterministic sanitized placeholders. These fixtures do not claim that the displayed properties are live or exhaustive.

The partial fixture mirrors the first two-source attempt: five final Crexi rows, zero LoopNet rows, and a deadline_exceeded source outcome. The complete fixture mirrors the final bounded run: fourteen raw rows, ten final rows, and exactly five rows retained from each source.

The public fixtures preserve only the evidence needed to exercise the contract. They do not contain private Actor identifiers, internal storage identifiers, real listing URLs, street addresses, broker details, credentials, or local filesystem paths.

Limits

This repository does not call an Actor. It only validates local evidence files, so it cannot prove that a future run will behave like either fixture.

It does not guarantee LoopNet availability. In the underlying experiment, several LoopNet transports failed before a later bounded run succeeded through a browser fallback. The complete fixture proves one run, not continuous source reliability.

It does not replace investment due diligence. The Actor works with public listing data, which can be incomplete, stale, or different from closed-deal records. Verify material facts against primary property and transaction sources.

It does not publish the proprietary scraper implementation. The repository contains only the agent-side validation contract, tests, and sanitized evidence shapes needed to reproduce the decision logic.

Security

• Prefer the hosted Apify MCP server with OAuth so project files contain no API credential.
• Scope the MCP URL to the Actor or tools required by the workflow.
• Set a strict per-run charge ceiling and a bounded result target before any execution.
• Never commit live run, build, dataset, key-value-store, cookie, contact, or local-path identifiers.
• Run npm run check:secrets and inspect git diff --cached before every push.
• Keep screenshots and unpublished editorial drafts outside this repository.

The privacy scanner rejects common credential and private-identifier patterns. It is a focused safeguard, not a replacement for a full security review.