Testing

Binding for implementation. This page defines the test layers, where each kind of test lives, the MIME conformance corpus, property tests and fuzzing, the integration harness against a local workerd with its fakes, fault injection and time control, the cross-tenant attack suite, the search and triage quality gates, the live end-to-end suite against staging, coverage, how every edge-case row maps to a test, and the CI checks. Rust workspace defines the xtask commands and the base CI pipeline; this page adds what they must contain.

1. Principles

1. A change ships with a test that fails without it (AGENTS.md definition of done).
2. Test where the rule lives. Pure rules are tested natively in core. Orchestration is tested natively against platform::fakes. Platform behaviour and wiring are tested against workerd. Only what needs real mail providers runs live.
3. Deterministic by default. Clock, randomness, DNS, models and vector search are fakes behind platform traits, so a failing test fails the same way every time.
4. No real data. Fixtures use RFC 2606 names (example.com, example.net, example.org, *.example, agents.example) and .invalid for the simulator; no real people, addresses or messages (CONTRIBUTING.md).
5. Names are contracts. Test names in the edge-case register exist verbatim in the code, and cargo xtask trace fails when one is missing (section 11).

2. Test layers

                      ┌─────────────┐  live::   staging, real Gmail/Outlook/SES        nightly, release
                    ┌─┴─────────────┴─┐ eval    real Workers AI + Vectorize            nightly, release
                  ┌─┴─────────────────┴─┐ it::, browser::  local workerd, fakes        every PR
                ┌─┴─────────────────────┴─┐ worker logic on platform::fakes (native)   every PR
              ┌─┴─────────────────────────┴─┐ conf:: MIME corpus (native)              every PR
            ┌─┴─────────────────────────────┴─┐ core:: unit + property tests, fuzz smoke every PR
            └─────────────────────────────────┘

3. Unit and worker-logic tests

• core takes time, randomness and lookups as arguments (Design conventions), so its tests pass fixed values: now_ms = 1_791_540_000_000, fixed 32-byte keys, fixed random bytes.
• core builds for wasm32-unknown-unknown too (CI wasm job); tests run natively.
• Worker-logic tests build a Platform bundle from platform::fakes: an in-memory clock that tests advance, a seeded RNG, rusqlite for D1 and Durable Object SQLite (with FTS5), maps for R2, captured queue sends, scripted AI and Vectorize, a DNS zone map, a recording HTTP client and a recording mail sender. They exercise transactions, outbox writes, state machines and policy without workerd.
• Error-code tests compare ErrorCode::http_status() and retryable() with every row of Errors (core::errors::catalogue_matches_reference), and every ErrorCode variant with the catalogue in both directions.
• Snapshot-style assertions compare JSON structurally (serde_json::Value), never as strings.

4. Property tests

proptest (pin at build time), native only. Each property runs 1,024 cases in CI and 65,536 nightly (PROPTEST_CASES).

5. MIME conformance corpus (conf::)

5.1 Layout

crates/conformance/
  corpus/
    mime/          b2_*, b4_*, b5_*, b6_*, b7_*, b8_*, b9_*, b13_*   structure, charsets, TNEF, nesting
    auth/          d1_*, d9_*                                      DKIM, ARC, DMARC, forged Authentication-Results
    dsn/           d4_*, g6_*                                      DSNs, MDNs, auto-replies
    threading/     c1_*, c2_*, c7_*, c8_*                          headers and subjects
    sanitize/      b7_*, b11_*, e1_*                               remote content, hidden text, injection text
    attachments/   b10_*, b12_*                                    risky types, archive bombs, extraction inputs
  dns/             zone files (TOML) for the fake resolvers: DKIM keys, DMARC and SPF records
  keys/            DKIM signing keys generated for tests only (file names end in .test-only.pem)
  golden/          the evaluation set (section 9)
  src/             loader, expectation checker, generators
  THIRD_PARTY.md   origin and licence of every imported fixture

Each case is a pair: <case>.eml and <case>.toml.

# crates/conformance/corpus/mime/b5_shift_jis_subject.toml
id      = "b5_shift_jis_subject"
edge    = ["B5"]
source  = "authored"                 # authored | generated:<generator> | derived:<origin> | imported:<project>
licence = "FSL-1.1-ALv2"

[envelope]
from = "sender@example.net"
to   = "bookings.acme@agents.example"

[expect]
subject        = "ご予約の確認"
flags          = []
text_contains  = ["予約番号 BK-2291"]
attachments    = 0
kind           = "normal"
verdict        = "none"              # with the zone fixtures in dns/

5.2 Sources and licensing

Never: real mail dumps, archives of public mailing lists, or anything containing a real person’s address. DKIM-signed fixtures are signed with the test-only keys and verified against the zone fixtures, so signatures are reproducible.

5.3 Runners

• Native (cargo test -p pylota-mail-conformance): parse each case with core, compare with [expect]. Test names are conf::<dir>::<id>, generated from the files, so the register’s wildcards (conf::mime::b2_*) match every case with that prefix.
• Through workerd (it::conformance::corpus_via_workerd): inject every case through the local email endpoint and compare the Message object returned by the API with the native expectations. This catches differences between native and wasm builds and gives the verdict-parity check of spike S4.

6. Integration tests against workerd (it::)

6.1 What cargo xtask itest does

As defined in Rust workspace, plus the details below:

1. Build the Worker with worker-build --release and the itest-hooks cargo feature.

2. Render deploy/wrangler.itest.toml: the production bindings, local resources, PM_ENV = "local", PM_PLATFORM_DOMAIN = "agents.example", PM_API_HOST = "localhost", PM_CONSOLE_HOST = "console.localhost" (the test client sends that Host header on console paths), PM_SIGNUP = "open", PM_WEB_BOT_AUTH = "on" (local only: the S13 gate applies to real deployments), the SES variables (PM_SES_REGION = "eu-west-2", PM_SES_INBOUND_*) and the Google, GitHub and Stripe client settings naming resources on the fake server, random test secrets written to .dev.vars in a temporary directory, PM_ITEST_FAKES_URL = "http://127.0.0.1:8798", a random PM_ITEST_TOKEN, queue consumers with max_batch_timeout = 1, and no AI or VECTORS binding (both are served by fakes). Tests inject provider events through the production Q_DELIVERY producer binding, which exists for dead-letter redrive.

3. Apply D1 migrations: npx --yes wrangler@4.139.0 d1 migrations apply pylota-mail --local --persist-to target/itest/state --config deploy/wrangler.itest.toml (a fresh directory per run). The rendered file sets migrations_dir = "../migrations/d1", because Wrangler resolves it against the file’s own directory, deploy/. Until v1.0 there is one file, 0001_init.sql.

4. Start npx --yes wrangler@4.139.0 dev --local --port 8799 --persist-to target/itest/state --test-scheduled --config deploy/wrangler.itest.toml, write its PID to target/itest/wrangler.pid, and capture stdout and stderr to target/itest/worker.log. Wait for GET /health.

5. Seed exactly what pmail setup writes in steps 19–22 (CLI and setup §6.3), in the same way:

   • one platform key (step 19, the bootstrap key) with wrangler d1 execute --local; the harness knows PM_KEY_PEPPER because it generated it;
   • the platform domain row for agents.example (step 20) with wrangler d1 execute --local, with records_json matching the DNS fake’s zone for it and monitor_do_id = ''. Until M13 builds the DomainMonitor, the row is written with state = 'healthy'. From M13 it is written pending, as setup writes it, and the harness triggers the every-minute cron (section 6.5), then advances the fake clock and runs the monitor’s alarm through /__test/alarm until it has verified the domain healthy;
   • the default tenant (step 21) through POST /v1/tenants with that key and address_suffix: "", because only the Worker can mint its TenantQuota ID;
   • from M6, which builds the minting hook, the system identity (step 22): its identities row (is_system = 1, mailbox_do_id = '') and primary address with wrangler d1 execute --local, then one every-minute cron run, after which its mailbox exists.

   Every other fixture (tenants, identities, domains, keys) is created through the public API.

6. Run cargo test -p pylota-mail-worker --features itest-hooks --test it -- --test-threads=1 with PM_ITEST_URL=http://127.0.0.1:8799. The it test target declares required-features = ["itest-hooks"], so cargo test --workspace never builds it.

7. Stop wrangler; delete target/itest/state unless --keep was passed.

6.2 Test hooks

Compiled only with itest-hooks, honoured only when PM_ENV = "local", and refused unless the request carries x-pm-test-token: <PM_ITEST_TOKEN>. cargo xtask build-worker refuses the feature and fails if the release bundle contains /__test/.

6.3 Fakes

All external services are served by one fake server inside the test process (crates/worker/tests/it/fakes/, a blocking HTTP server on 127.0.0.1:8798; the HTTP server crate is pinned at build time). With itest-hooks, platform::itest provides implementations of the platform traits that call it:

• Test tenants use the real simulator and loopback code (*@simulator.invalid, L2, L3); only live tenants use the fake mail sender. The real Cloudflare transport is exercised by spike S1 and the live suite: the local send_email simulation cannot serialise binary attachments (Cloudflare Email Service local-development docs, read 2026-10-09).
• Webhook receiver fakes record each request (headers, body, signature check result) and can return any status, delay past the 15 s timeout, redirect, or stream an oversized body.
• The SSRF guard runs unchanged: the DNS fake answers webhook and SMTP relay hosts with a fixed public address that is never contacted, because the fake HTTP client and the socket fake route by host name.
• The groups that use these fakes: it::ses::* (SNS push, SQS, S3 and SES fakes), it::smtp::* (the SMTP server fake), it::forwarding::* (the mail sender fake plus inbound injection at the platform address), it::domains::* (DNS, Cloudflare API and SES fakes), it::oauth::* (OAuth fakes and one cookie jar per simulated browser), it::checkout::* and it::signup::* (Stripe fake), it::totp::*, it::landing::*, it::onboarding::* and it::abuse::* (fake clock), it::hosts::* (the two Host values), it::identity_keys::*, it::assertions::*, it::http_signatures::* and it::well_known::* (fake clock for overlap windows and expiry; the Rust SDK’s verify_assertion runs natively in the test process against the JWKS that workerd serves), and it::notify::* (fake clock for holds, windows, the 09:00 run, time zones and cooldowns; /__test/alarm with class notifier; notification emails are observed the same way as console sign-in mail, which the system identity also sends (Console › Requesting a link or code); /__test/delivery-event for a hard bounce on a notification; the DNS fake to make the platform domain failing).
• Another value of a deployment variable or secret. A test that needs one (PM_CONSOLE=off for it::console::disabled, PM_WEB_BOT_AUTH=off for it::http_signatures::disabled_and_policy, PM_BILLING=off for it::notify::billing_off_no_usage_alerts, PM_NOTIFICATIONS=off, or the secret PM_MASTER_KEY_NEXT for the master-key rotation tests) calls restart_runtime_with(&[(name, value)]), which restarts wrangler like restart_runtime() (section 6.6) with the value overridden in the rendered wrangler.itest.toml or .dev.vars, and restores the original on exit.

6.4 Injecting inbound mail

The default path is the endpoint wrangler dev provides for email handlers (Cloudflare docs, read 2026-10-09): POST http://127.0.0.1:8799/cdn-cgi/local/email?from=<envelope from>&to=<envelope to> with the raw RFC 5322 message as the body; the message must have a Message-ID header. One call per envelope recipient, as Email Routing invokes the handler once per recipient (A9). The documentation does not say how a setReject is reported to the caller, so tests read the outcome from /__test/inbound or from the inbound_rejected log line; S1 records the endpoint’s actual response.

6.5 Time control

• Platform clock. platform::itest::Clock returns Date.now() + offset, with the offset set by POST {fakes}/clock { "advance_ms" } or { "set_ms" } and synchronised at each invocation (6.2).
• Alarms. Objects arm alarms at absolute times computed from the fake clock, which may be far in the real future; tests run them with /__test/alarm. Purposes not yet due at the fake clock do not run.
• Cron. GET /cdn-cgi/local/scheduled?cron=<expression>&time=<ms> triggers scheduled() with that cron and scheduledTime (Cloudflare docs, read 2026-10-09); time is the fake clock.
• Queue delays. platform::itest producers and Incoming::retry record the requested delay at the fake server (GET {fakes}/queue-log) and send with min(requested, 1) second. Retry-schedule tests (J4, G3, G8) assert the recorded delays.
• Waiting. Asynchronous effects are awaited by polling the API with backoff (50 ms doubling to 1 s) for at most 20 s; a test never sleeps a fixed time.

6.6 Fault injection

POST {fakes}/faults arms a fault plan; decorators in platform::itest consult it before each call:

{ "target": "r2.put", "match": { "key_prefix": "t/" }, "mode": "error", "count": 3 }

Runtime restarts. For J2, the harness helper restart_runtime() kills the wrangler process from target/itest/wrangler.pid mid-test and starts it again with the same --persist-to directory, then asserts the queue retry produced exactly one stored message.

6.7 Isolation and logs

• Each test creates its own tenant (slug = "t" + 10 hex of the test name's hash), so tests do not see each other’s data. Tests that change global state (clock, fault plans, platform domain records) reset it in a guard on exit; --test-threads=1 keeps them serial.
• The I5 log-scrubbing test (I5) runs last: it reads target/itest/worker.log and fails if any canary string appears. Every fixture plants canaries: a unique token in each body, subject, display name, filename and attachment text, and every address used by the suite, plus every key and webhook secret the suite created.

6.8 Browser suite (browser::)

From M21 on, cargo xtask itest runs the Playwright suite in crates/worker/tests/browser/ after the it:: tests, while wrangler and the fakes are still running, with npx --prefix crates/worker/tests/browser playwright test (Node.js 22 and the Chromium build of the pinned Playwright release, which the CI job installs). --suite it and --suite browser run one suite alone. The test titles carry the browser:: names, which cargo xtask trace collects like the Rust ones.

Pages added by M24 (sign-up, two-step verification, the Overview) and M26 (notification settings and the unsubscribe pair) join both tests when they land.

6.9 Benchmarks

The it::bench::* tests that need a large mailbox (it::bench::keyword_p95 and it::bench::hybrid_p95 on 50,000 messages, and it::bench::tenant_fanout_p95 over 10 identities) fill it with POST /__test/bulk-seed and then measure through the public API. They are marked #[ignore], so the pull-request itest run skips them, and the nightly workflow runs them by passing --ignored to the test binary. Each reports its figure and warns above its target. They are never a required check on a pull request: a 50,000-message seed takes minutes, and timings on shared CI runners are noisy.

7. Cross-tenant attack suite

Proves NFR-SEC-1 (zero cross-tenant access), FR-KEY-3 and the partner isolation of FR-KEY-4. Rules are in Security › Authorisation.

Fixture. Two tenants, A (victim) and B (attacker), each with two identities, a domain, a webhook, a key of each level holding every permission valid at that level, threads with messages and attachments, a held thread, an erasure request, an export, an identity signing key on each identity (one rotated, so a retiring key exists too) and policy.web_bot_auth.allowed = true. Tenant A’s mail contains a unique canary term. A third tenant C is a test tenant. Two partners, P and Q, each have a partner key holding every permission valid at the partner level and a partner webhook endpoint: P’s key created A, Q’s key created B, and C was created by a platform key, so it has no partner. A’s domain was added with nameservers, so the Cloudflare fake holds A’s zone and its zone_claims row; the fake also holds the zone of PM_PLATFORM_DOMAIN.

“Every permission valid at that level” follows Security §4.6: a tenant key holds every permission except tenants:manage, partners:manage and platform:ops, so it holds identities:sign; an identity key holds the same set without the tenant-only permissions (members:read, members:manage, suppressions:manage, audit:read, usage:read), plus usage:read implicitly for its own workspace. A partner key holds every permission except platform:ops, partners:manage and identities:sign.

Attacker key classes (each with full permissions for its level):

Matrix. it::security::cross_tenant_matrix reads GET /__test/routes and, for every route with a path parameter and every attacker class, calls the route with A’s resource IDs (and, for POST/PATCH, a valid body). The enumeration includes the identity-key routes (…/keys, …/keys/rotate, …/keys/{kid}/revoke with A’s kid), POST …/assertions and POST …/http-signatures on A’s identities; the scope check answers before any signing rule, so they give the same 404 identity_not_found as a missing identity. For each call it also makes a control call with the same key and a random non-existent ID of the same type. It asserts:

1. The status is 404 with the route’s *_not_found code, or 403 scope_denied for a route above the key’s level on its own tenant, or 401 for revoked and expired keys.
2. The attack response equals the control response byte for byte, except request_id and the Request-Id and RateLimit-* headers (indistinguishability).
3. No side effect: D1 row counts for A, A’s mailbox state (via a platform key), A’s outbox and A’s webhook receiver are unchanged, and no audit_log row names A.
4. The same matrix runs over MCP: every tool, with the same attacker keys, returns an error and no data.

Zone case foreign_zone (H8). B’s tenant key and Q’s partner key call POST /v1/tenants/{B}/domains with method: "cloudflare_zone" (once with replace_mx: true) for A’s zone, for a name under it, and for a name under the platform domain’s zone, also after a platform key lists A’s zone in B’s domains.cloudflare_zones; and with nameservers and delegated_subdomain for a name under either zone. Each gets 403 scope_denied with details.reason = "zone_not_allowed", the same body as for a zone that does not exist, before any call reaches the Cloudflare fake; no D1 row is written, no MX record of A’s zone is deleted, and A’s routing is unchanged.

Additional suites.

The suite is part of cargo xtask itest and therefore a required check on every pull request. Timing is not asserted in CI (too noisy); both code paths do the same D1 read by construction.

8. Fuzzing

The fuzz project lives in fuzz/ (cargo-fuzz, libFuzzer, nightly toolchain only) with the targets listed in Rust workspace. The five required by the edge-case work:

• Seeds come from crates/conformance/corpus/.
• CI runs the five for 60 seconds each on every pull request (fuzz-smoke); nightly runs every target for 10 minutes.
• A crash is minimised (cargo fuzz tmin), committed as a regression input under crates/core/tests/fuzz_regressions/<target>/, and replayed by core::fuzz_regressions::replay_all in the normal test run.
• Before a release every target must have run clean for 24 cumulative hours on the release commit (Security).

9. Search and triage evaluation

The golden set, the labelled queries, the agentic questions, the triage labels and the metric definitions are owned by Search › Quality evaluation and Triage › Evaluation set. This section defines how the harness runs them.

9.1 Files

crates/conformance/golden/
  generator.toml    seed and template mix; the mailbox (about 5,000 messages in four identities of
                    tenant `acme`) is generated deterministically at run time and never committed
  hard_cases/       hand-written .eml files added to the generated mailbox
  queries.toml      labelled queries with graded relevance per message key
  questions.toml    agentic questions with gold facts and gold supporting message keys, including
                    unanswerable and steering questions
  triage.toml       labelled triage set

Message keys are stable names (brightwell_invoice_88213) mapped to message IDs at load time. All content is synthetic, on reserved domains, under the repository’s licence (FSL-1.1-ALv2). Accepted scores (the baseline) are recorded in docs/src/project/quality.md, as the search and triage designs specify.

9.2 Running

cargo xtask eval-search, eval-agentic and eval-triage:

1. Start wrangler dev without --local, with the AI binding (always remote) and the Vectorize binding set to remote = true against a dedicated index pm-mail-chunks-eval in the CI Cloudflare account; D1, R2, Durable Objects and queues stay local. If the pinned Wrangler cannot bind Vectorize remotely, the Worker uses the Vectorize REST fallback with PM_CF_API_TOKEN (Rust workspace).
2. Generate the golden mailbox and inject it through the local email endpoint (section 6.4); wait until semantic_coverage = 1.0 for every identity.
3. Run every query, question or labelled message through the public API (triage through POST …/messages/{id}/triage); write target/eval/<suite>.json with per-item results and totals.
4. Compare with the baseline in quality.md and fail on a gate.

9.3 Metrics and gates

Pull requests run the same pipelines with the scripted fake model, so prompts, fencing, budgets, the verifier and schema validation are checked without network access. In addition, it::search::golden_keyword_recall loads the golden mailbox with the fake AI inside cargo xtask itest and asserts that keyword recall@10 does not drop against the baseline: keyword search is deterministic, so this is a required pull-request check. Updating the baseline is a reviewed change with the score deltas in the pull request description.

10. Live end-to-end suite (live::)

PRD release criterion 3. cargo xtask live runs cargo test -p pylota-mail-worker --features live --test live -- --test-threads=1 against the staging deployment (its own zone, platform domain, D1, R2, Vectorize and queues, Architecture).

Manual tests. Rows marked Manual need a person in a browser (a Google consent screen, a Stripe Checkout page, Gmail’s unsubscribe button). They are #[ignore]d in the nightly run. Before a release, a person runs cargo xtask live --manual against the release candidate on staging: the harness runs the scripted parts, prints each browser step and waits for the person to confirm it, then checks the outcome through the API exactly as an automated test would, and records the result per test in target/live/manual.json. The release job needs a passing manual run on the release commit (section 12).

Staging configuration. Staging runs with PM_BILLING=stripe, Stripe test-mode keys and its own PM_PLAN_CATALOG (deploy/staging/plan-catalog.json): the production plan names with Stripe test-mode price IDs and small allowances (Free 10 sends, Developer 20 sends, a sends top-up of 5 units), so step 11 reaches 402 and step 15 crosses 80% within a few sends.

Secrets. Live tests read credentials only from the GitHub Environment staging, which requires a reviewer and is limited to main and release tags; forks never receive them. The environment holds: a Cloudflare API token scoped to the staging account, a staging platform key with a 90-day expiry, OAuth credentials limited to the two dedicated test mailboxes (Google Workspace and Microsoft 365, holding only synthetic mail), AWS credentials for the staging SES resources, and an API token for the external DNS provider that hosts the dns_records test domain. The secret names are listed in Build plan › Human prerequisites (STAGING_*). The harness never prints secrets, redacts them from failure output, deletes test messages from the mailboxes after each run, and the credentials are rotated every quarter.

Schedule. Nightly, and on every release candidate before the production rollout (section 12).

11. Edge-case mapping and coverage

11.1 Naming

• The row ID (a6, j7) starts the last segment, so cargo test a6_ finds every test for a row. A test that covers several rows, or a requirement rather than one row, may omit it (for example it::ses::retired_rule_sync covers N7 and N29); the register names it explicitly.
• A * in the register (conf::mime::b2_*, it::send::g1_*) means at least one test with that prefix.
• Rows owned by I (integrator) have no service test. Rows owned by S+I have the service-side test named in the register.
• Every test function carries a doc comment line Covers: <IDs>, for example /// Covers: FR-OUT-1, G1.

11.2 cargo xtask trace

Parses docs/src/project/edge-cases.md and docs/src/project/prd.md, collects test names and Covers: lines from the source tree (including generated conf:: names), and fails when:

• a test named in an S or S+I row does not exist (PRD release criterion 2);
• a P0 requirement has no test with it in Covers: (PRD release criterion 1);
• a Covers: line names an unknown row or requirement.

It prints the traceability matrix as Markdown into the CI summary.

11.3 How rows are exercised

11.4 Coverage

• cargo llvm-cov (cargo-llvm-cov, pin at build time) runs on cargo test --workspace in CI.
• pylota-mail-core must keep line coverage at or above 85%; the job fails below it. Other crates are reported, not gated.
• Coverage never replaces the traceability check: a covered line without a named test for its rule is not “tested”.

12. CI workflows and required checks

The base jobs are those in Rust workspace › CI pipeline. This design adds the security and traceability jobs:

Required checks to merge into main: fmt, clippy, test, layering, wasm, itest, browser, fuzz-smoke, deny, audit, openapi, docs, trace, codeql. The milestone gate’s cargo xtask itest runs both suites that the itest and browser jobs split.

Required to publish a release: all of the above on the tagged commit; the three evaluation gates (section 9.3); the live:: suite green on staging with that commit deployed, including a passing manual run (cargo xtask live --manual, section 10); no open crash from the nightly fuzz run; the security pre-release checklist.

GitHub Actions are pinned to full commit SHAs and each job declares least-privilege permissions (Security › Supply chain).

13. Tests of the test infrastructure