Verification and tests

The test suite

tests/ — 133 tests, all runnable offline with fake agents and fake LLMs (no network, no models, no mail servers):

cd /path/to/this-repo
pip install -r ralph_orchestrator/requirements.txt pytest
/opt/homebrew/opt/python@3.11/bin/python3.11 -m pytest tests/ -q     # or: python3 -m pytest tests/ -q

Verified on this repository: 133 passed (Python 3.11).

File Covers

test_orchestrator.py

the run loop end-to-end with a fake shell agent: parallel waves and dependency visibility, scope-overlap serialisation, regression-failure merge undo, resume skipping merged tickets, blocked ticket stopping its dependents, in-run IMAP polling (silent no-op when unconfigured, sender-account fallback, mid-run reply → feedback, one-shot report flip, never raises into the loop), operator-reply unlock of a blocked ticket (including re-running it), fresh-repo state creation

test_graph.py

waves, critical_length and suggested_agents computation; validate (cycles, unknown dependencies, duplicates, empty accept/scope); in_scope / scopes_overlap glob helpers

test_consensus.py

parse_verdict (think blocks, code fences, garbage), decide (quorum, abstentions, no valid votes), Panel.vote with a fake model factory including skip_author_model

test_collab.py

bus post/inbox/cursors; change requests (approved → fix ticket created and dependents rewired, rejected, ask auto-answer); a full run that includes a fix ticket; contract negotiation (conflicts drive a corrected round, force_resolve owner decision wins)

test_planner.py

the planner repair loop: an invalid plan (unknown dependency, vacuous accept) followed by a corrected one; think-block and fenced-JSON extraction; output round-trip through load_tickets

test_mail_reply.py

Message-ID stability per run; matches_reply (In-Reply-To, References, subject fallback, unrelated mails rejected); _own_text quote stripping; fetch_replies (never raises when unconfigured, sender-account fallback, bad server); record dedup by id; file import dedup; consumption (feedback on mentioned tickets, general replies leave feedback untouched, per-ticket reply events)

test_mail_config.py

the mail: → env shim (fills only empty vars, existing env wins), empty mail: defaults, gateway key precedence (env > yaml > file), mail_check.load_saved (config.yaml beats the legacy start.env)

test_report.py

reason-keyed sign-offs, the attachment policy (final/manual mail carries HTML and PDF, periodic stays plain text, one-shot pdf_only after a report reply), the word-boundary report trigger, flag non-consumption, last_report.txt writing

test_html_report.py

snapshot building (counts, completion %, distinct statuses, stale heartbeat, real dependencies only, hierarchical overlap-free layout), state events and old-format backfill, self-contained report/dashboard HTML (no CDN), the dashboard HTTP server, the Timeline view, the watch badge, the plugins box, empty-state and large-task-set rendering, script-injection safety

test_pdf_report.py

PDF rendering of a real report via Playwright Chromium (skipped when the browser is not installed), write_run_report with and without PDF, the HTML-only fallback, print CSS, actionable-task pre-selection, the Markdown appendix

test_install.py

ensure_opencode_plugins (writes project plugin config, preserves existing entries, no-op without plugins, survives installer failure), cmd_install (wires the npm plugin and runs the one-time caveman installer, skips it when present), resolve_plugins (project/global detection, missing-expected flag, dedupe), the plugins box in report and dashboard, plugins in the snapshot

test_watch.py

rework (pending only, merged/blocked history preserved), the watch cycle (unlock + rework, yields to a live run, idle when everything is merged), live_run_active (requires a live pid and a fresh heartbeat)

test_wizard.py

wizard_problems (missing bits, clean config), parse_opencode_providers, endpoint probes (fallback when the gateway lacks /models, reachable / key-rejected / unreachable), wizard paths (banner + stop, missing CLI with unreachable provider, happy path, exhausted stdin)

Test fixtures use temporary git repositories (real git init/commit in tmp_path), fake agents as shell case scripts, and a make_llm factory substitute returning scripted verdicts — so the orchestration logic is tested exactly as it runs in production, without any external system.

End-to-end checks (reproducible)

  • The showcase — a full real run of the orchestrator against the dummy agent, 4 tickets, parallel wave, no LLM: 2-minute offline showcase (verified output: 4/4 merged, peak agents: 2, reason: DONE).

  • The dashboard — after any run:

python3 ralph_orchestrator/orch.py dashboard --port 8471 &
curl -s http://127.0.0.1:8471/api/run | python3 -m json.tool | head    # snapshot JSON
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8471/         # 200

  • The kit install — ./install.sh /path/to/a/git/repo into a scratch repository: the kit appears, .venv-ralph is created, .ralph/ and .venv-ralph/ are in info/exclude, and orch.py check runs against a hand-written tickets.yaml.

Known limitations

  • The consensus/planner/contracts paths are tested with fake LLMs; live gateway behavior (key rotation, model outages) is covered by preflight --ping and the wizard’s reachability probe at setup time.

  • The PDF step requires Playwright Chromium; without it the run degrades to HTML only by design (tested).

  • IMAP fetching is tested with canned RFC822 payloads and header parsing; no live mail server is exercised in the suite — verify with orch.py mail-check.