E-mail loop and watch agent

The e-mail layer is the operator channel: the run talks to you on a schedule, and your replies come back into the loop as ticket feedback — including unlocking blocked tickets. It is best-effort end to end: a failing mail or IMAP fetch is printed and swallowed, it never stops a run.

Configuration and setup

All settings live in the mail: section of config.yaml (configuration.adoc#mail) or in environment variables, which win when set. Verify before a long run:

python3 ralph_orchestrator/orch.py mail-check

mail_check.py walks the connection step by step — DNS, TCP, TLS, login, send — with a diagnostic hint per failure class (host not found / port blocked / TLS mismatch / authentication), asks the password hidden, sends a test mail, and exits 0 only when the server accepted it. Standalone: python3 ralph_orchestrator/mail_check.py [--from-env] (defaults: environment → config.yaml mail: → legacy .ralph/start.env; the password is never saved).

The mails

report.py composes and sends. Every mail carries a stable per-run Message-ID: <ralph.<run-start-epoch>@<hostname>> (derived from the run’s recorded start time, never the pid) — replies are matched against it.

Mail When Body Attachments

Mail	When	Body	Attachments
Periodic check-in	every `report.every_seconds` (default 600 s) while the run is active	plain text: greeting, a progress line built only from the real counters ("I am 44% done (4 of 9 tickets merged). 1 agent is working very hard…"), signature, reply how-to	none — until you reply "report" (see below)
Final report	when the run stops (any reason)	reason, run time, agent runs, peak agents, the wave plan, one line per ticket (status/attempts/model/title), ACTION NEEDED block with the last feedback of blocked tickets, which report files are attached, reply how-to, a reason-keyed sign-off	`out/run_report.html` + `out/run_report.pdf` when it exists
Manual	`orch.py report`	final-report body	same as final

Periodic check-in

every report.every_seconds (default 600 s) while the run is active

plain text: greeting, a progress line built only from the real counters ("I am 44% done (4 of 9 tickets merged). 1 agent is working very hard…"), signature, reply how-to

none — until you reply "report" (see below)

Final report

when the run stops (any reason)

reason, run time, agent runs, peak agents, the wave plan, one line per ticket (status/attempts/model/title), ACTION NEEDED block with the last feedback of blocked tickets, which report files are attached, reply how-to, a reason-keyed sign-off

out/run_report.html + out/run_report.pdf when it exists

Manual

orch.py report

final-report body

same as final

A one-shot report attachment: when an operator reply contains the word report, the next periodic check-in attaches the printable PDF only (no HTML copy — pdf_only), and the reply is flagged report_delivered so subsequent check-ins go back to plain text.

Delivery (send):

SMTP — MAIL_TO + SMTP_HOST required; port 587 = STARTTLS (unless SMTP_STARTTLS=0), port 465 = implicit SSL; login with SMTP_USER/SMTP_PASS when both are set. From = SMTP_FROM or SMTP_USER.
Webhook (optional) — REPORT_WEBHOOK receives {"text": "<subject>\n\n<body>"} (JSON POST, 30 s timeout); text only, no attachments.
The full subject/body/Message-ID of every report is always written to .ralph/last_report.txt even when nothing is sent.

Reply capture

mail_reply.py finds your answers in the sender’s IMAP inbox (replies to the report mail arrive at the sending account; IMAP_USER/IMAP_PASS default to SMTP_FROM/SMTP_USER
SMTP_PASS).

sequenceDiagram participant O as operator (mail client) participant I as IMAP inbox (sender) participant R as Ralph (run / watch / mail-reply) participant X as .ralph/inbox.json participant S as state.json O->>I: reply to the status mail
(In-Reply-To: @>) R->>I: every report.reply_poll_seconds (run) or each cycle (watch):
SEARCH SUBJECT "[ralph]" (server-side) I-->>R: matching UIDs (capped at --limit, default 100) R->>I: fetch RFC822.HEADER per UID R->>R: matches_reply: In-Reply-To / References contains our Message-ID
(fallback: subject "Re: [ralph]" / "Fwd: [ralph]") R->>I: fetch full RFC822 for the actual matches only R->>X: append {id, from, date, subject, body}, deduped by message id, seen=false R->>S: consume_replies (below)

The server-side subject search (not "the most recent N") is deliberate: a busy inbox can outpace a fixed window, and Gmail keeps sent mail in INBOX, so the sender’s own report mails match the search too — matches_reply keeps only the real answers.

Offline path: orch.py mail-reply --file reply.txt [--subject "me"] imports a saved reply (e.g. copied out of a mail client) into inbox.json, deduped by a content hash — re-importing the same text does nothing.

Consuming replies

consume_replies runs at the start of every orch.py run, on the run’s in-loop IMAP poll (In-run polling), and on every watch cycle. It never raises. For each unseen inbox entry:

The reply is appended to state.data["replies"] (capped at 50) and recorded as a reply event — visible in the dashboard timeline and the run report.
The body is reduced to what you actually typed (_own_text): everything from the first quoted line (`> `) or quote marker ("wrote:", "ha scritto:", "Original Message") is dropped — otherwise the check-in’s own example text ("T6: retry with a covering index") would flag T6 on every plain "report" reply.
Every ticket id word-boundary-mentioned in your text is known tickets:
- a reply event is recorded on the ticket,
- your full reply is appended to its feedback — the worker’s next attempt sees it under Feedback from the previous attempt,
- if the ticket is blocked it is unlocked: status back to pending, attempts reset to 0 — so your instruction is actually acted on (mid-run on the next wave, or at the start of the next run / by the watch agent).
The inbox entry is marked seen with consumed_at.

Practical example — a blocked ticket, unblocked by mail:

# the run ends BLOCKED; the final mail lists:
#   ACTION NEEDED: blocked tickets (last feedback):
#   T4: The acceptance command failed. Output (tail): KeyError: 'rate'

# you reply to the mail:
#   "T4: the key is 'exchange_rate', not 'rate'. Use it in prices.py."

# within 30 s (in-run poll) or on the next run/watch cycle:
#   T4: unlocked (attempts 0) with your reply appended to its feedback
#   -> the next attempt for T4 sees: "T4: the key is 'exchange_rate', ..."

In-run polling

While a run is active, Orchestrator.poll_replies fetches + records + consumes replies every report.reply_poll_seconds (default 30 s) and regenerates the run report so the dashboard timeline shows your reply in real time. It is a quiet no-op without IMAP_HOST and a resolved sender account.

The watch agent (correction agent)

orch.py watch stands by between runs for the gaps the in-loop poll cannot cover: a blocked ticket you want fixed later, without re-driving the whole loop.

nohup .venv-ralph/bin/python ralph_orchestrator/orch.py watch &

Each cycle (--once for a single pass):

If a live run drives the repo (state pid alive and heartbeat < 120 s), the cycle is a no-op — the run handles e-mail itself.
Otherwise: fetch replies from IMAP (when configured), record + consume them, and — if any ticket changed status or any ticket is pending — run a bounded rework pass: only pending tickets are relaunched, merged/running history is preserved, no final reason, no mail (orchestrator.adoc#re-work).
Every cycle heartbeats .ralph/watch.json (pid, heartbeat, note).

The dashboard shows a watch standing by badge while that pid is alive and fresh (Dashboard and run report). Ctrl-C stops the watch cleanly.