Consensus review panel

consensus.py implements the review gate: a panel of independent LLM reviewers votes on a change. It is used in three places — ticket review after acceptance, contract approval (collaboration.adoc#contracts), and change-request adjudication (collaboration.adoc#change-requests) — all through the same Panel.vote primitive.

Panel composition

The panel is llm.reviewers from config — a list of {name, path, model, author_id} specs, each bound to a different gateway model (configuration.adoc#llm). When consensus.skip_author_model is enabled (default), the reviewer whose author_id matches the ticket’s agent model is excluded — a model never reviews its own ticket (if that would empty the panel, the full list is used).

Voting mechanics

Panel.vote(system, body, author_id):

  1. Every reviewer gets the same prompt (a system prompt + the body) as a LangChain chain: ChatPromptTemplate | make_llm(cfg, spec) | StrOutputParser | parse_verdict.

  2. Chains run in parallel (RunnableParallel); each has a fallback to None — a reviewer that errors or returns garbage abstains, it does not fail the vote.

  3. parse_verdict extracts the JSON verdict from the reply: strips think blocks, takes the outermost {…​} (or the first valid embedded object), and requires a boolean approve field. Unparsable replies abstain.

  4. decide(verdicts, quorum) applies the rule below.

flowchart LR A[body: criteria + scope +
acceptance output tail + diff] --> B[reviewer 1
model X] A --> C[reviewer 2
model Y] A --> D[reviewer 3
model Z] B -->|JSON verdict| V[decide] C -->|JSON verdict or abstain| V D -->|JSON verdict or abstain| V V -->|approvals ≥ min(quorum, valid) and valid ≥ 1| OK[approve] V -->|otherwise| NO[reject +
every issue listed]

The decision rule

decide is a veto-free majority rule:

  • approvals must reach min(quorum, number of valid votes) — with quorum: 2 and only one usable reply, that single approval suffices;

  • at least one valid vote must exist — if every reviewer abstains the change is rejected with Review panel unavailable: no reviewer returned a usable verdict.

  • a rejection reports every reviewer’s verdict (name: APPROVE/REJECT - summary) and, on failure, every issue from every valid vote, so the next attempt gets actionable feedback.

The ticket-review body (HUMAN prompt) contains: ticket id/title/description, the criteria list, the scope list, the last 4000 characters of the acceptance command output (already executed by the orchestrator), and the diff against the integration branch, truncated at consensus.max_diff_chars (default 60 000, with a [diff truncated] marker). The reviewer system prompt (SYSTEM) instructs: judge only from the evidence; reject if criteria are not really met, behaviour is hard-coded to pass a test, tests were weakened or skipped, files outside the scope were touched, functionality is missing, or existing code/style/framework was reinvented.

Verdict shape

{"approve": true|false, "issues": ["concrete, actionable problem", ...], "summary": "one sentence"}

parsed into Verdict(approve, issues, summary).

Tuning

  • consensus.enabled: false (or orch.py run --no-consensus) skips the gate entirely — the acceptance command alone then guards the merge.

  • consensus.quorum — required approvals (capped by valid votes).

  • llm.reviewers — add/remove reviewers or change their models; each path is a gateway model path.

  • The rule itself lives in consensus.py:decide — change it there.