Agentic Software Factory · 2026-07-17 · This or That: RLHF for Your Agent
Session Replay: the loop runs on itself
On the eve of the workshop, the this-or-that kit was pointed at its own question — how should these guides be written? Thirteen formats, 27 votes, about ten minutes, and one typed note that reshaped every section's pages. Including this one.
The session
This transcript restages a prior working session in beats. Every vote line, the standings table, and the quoted note are verbatim from the round's vote record (archived as authoring-lab/votes-round1.jsonl) and its RESULTS.md.
I want the workshop guides to move away from numbered exercises toward write-ups grounded in what actually happened in past sessions — user stories, skills, something in that direction. I can't describe the format I want. Run the this-or-that loop on it: the question is how should workshop projects be authored?
The classic opening condition for this loop: a real decision, more than one plausible answer, and taste the deciding factor the owner can't articulate.
AgentFramed the axis before generating anything. One thing varies: content architecture. Everything else held constant — every option renders the same subject (another section's custom-zsh-functions material) through one shared stylesheet, with light/dark supported so the mode lands on every vote. Then scaffolded authoring-lab/ at the repo root: compare.html and server.py copied verbatim from this section's kit, plus the manifest — 13 concepts, incumbent included as the control:
{
"title": "How should workshop projects be authored?",
"modes": true,
"options": [
{ "id": "exercise", "name": "Exercise (incumbent)", "src": "options/01-exercise.html" },
{ "id": "user-story", "name": "User Story", "src": "options/02-user-story.html" },
{ "id": "session-replay", "name": "Session Replay", "src": "options/03-session-replay.html" },
{ "id": "scar-story", "name": "Scar → Fix", "src": "options/04-scar-story.html" },
{ "id": "skill-card", "name": "Installable Skill", "src": "options/05-skill-card.html" },
{ "id": "before-after", "name": "Before / After", "src": "options/06-before-after.html" },
{ "id": "case-file", "name": "Case File", "src": "options/07-case-file.html" },
{ "id": "prompt-anthology", "name": "Prompt Anthology", "src": "options/08-prompt-anthology.html" },
{ "id": "decision-journal", "name": "Decision Journal", "src": "options/09-decision-journal.html" },
{ "id": "dialogue", "name": "Socratic Dialogue", "src": "options/10-dialogue.html" },
{ "id": "field-manual", "name": "Field Manual", "src": "options/11-field-manual.html" },
{ "id": "run-of-show", "name": "Run of Show", "src": "options/12-run-of-show.html" },
{ "id": "agent-brief", "name": "Agent-First Brief", "src": "options/13-agent-brief.html" }
]
}
The kit teaching this section is the kit being used — copied, not rebuilt. And 13 is deliberately past the comfort zone: structural distinctness is what makes votes cheap. If two options could be confused at arm's length, one gets replaced.
AgentAuthored one exemplar option directly, then fanned out parallel subagents, each writing 2–3 formats from one tight shared brief: subject digest, format rules, per-option spec, the exemplar's path. All 13 render the same zsh material; only the architecture differs — an annotated session transcript, a Socratic Q&A, an investigation case file, a time-coded run of show, an installable skill, an agent-first brief, the incumbent exercise…
QA before handover: every option HTTP 200 through server.py; grep audit for external requests, absolute home paths, secrets; both modes rendered; one smoke vote POSTed to /vote, confirmed in votes.jsonl, then removed. Server started in the background — here's the URL. Controls: ←/→ pick · ↓ both bad · d flips light/dark · s skip · f finish. The note field saves with each vote.
The handover ends with "one note outweighs ten votes" — about to be proven — and then the agent gets out of the way. Hovering ruins honest voting.
You[arrow keys, 27 duels, about ten minutes by the timestamps]
Meanwhile, every keystroke lands as a full record. Vote 7 as it hit votes.jsonl (Elo snapshot elided here — the real line carries all 13 ratings):
{"mode": "elo", "vote": 7, "left": "Installable Skill", "right": "Session Replay",
"winner": "Session Replay", "loser": "Installable Skill",
"elo": { …all 13 ratings after the vote… },
"note": "", "ui_mode": "light", "lab": "How should workshop projects be authored?"}
Session Replay beat the Installable Skill twice (votes 7 and 20) — that head-to-head became a banked taste rule: show the session, not just the recipe. Past-session evidence outranks the same content written as instructions.
YouDuel 26 — the top two meet head-to-head, Session Replay vs Socratic Dialogue. Instead of just picking, a note goes into the field:
{"vote": 26, "left": "Session Replay", "right": "Socratic Dialogue",
"winner": "Socratic Dialogue", "ui_mode": "dark", "toggled": true,
"note": "I love _both_ the \"session replay - custom zsh functions\" and
\"dialogue - custom zsh functions\". I think we could use both in
separate pages so that each project has a session replay and a
dialogue page"}
The decisive artifact of the whole round is one typed sentence. The ranking alone would have shipped Dialogue and discarded Replay. A note that makes a decision is the decision.
YouDone — read the votes.
Read votes.jsonl back: 27 votes, zero both-bad, per-mode splits on every line. Final standings:
# Format Elo W-L Light Dark
1 Socratic Dialogue 1064 4-0 2-0 2-0
2 Session Replay 1048 4-1 2-0 2-1
3 Agent-First Brief 1031 3-1 2-0 1-1
4 Exercise (incumbent) 1016 3-2 1-2 2-0
5–9 User Story · Scar → Fix · Installable Skill ·
Prompt Anthology · Run of Show all 1000, 2-2
10–12 Before/After · Decision Journal ·
Field Manual all 968, 1-3
13 Case File 937 0-4 0-1 0-3
Stated the taste as rules with the evidence attached: conversation beats document — the top three all read as turns addressed to someone, and every dossier-shaped format sank. Show the session, not just the recipe. Numbered steps can live inside a page but can't be its identity — the incumbent finished 4th, outclassed rather than hated. Bureaucratic chrome kills — case numbers and PROC blocks took the bottom slots, and Case File went 0-3 in dark mode alone.
All of it banked: standings, the quoted note, and the taste rules written to RESULTS.md; the raw record archived as votes-round1.jsonl. Decided once, with evidence — never re-litigated.
AgentThen the loop itself got captured, so the next round costs one command instead of a re-explanation: an mdflow flow at flows/this-or-that.md that runs the whole process and injects the accumulated taste records fresh on every run, and a generalized agent skill at ~/.agents/skills/this-or-that/SKILL.md that pulls this same kit from this site's /kit/ files for any repo. Both are on the section guide, flow file included verbatim.
The outcome is what you're reading. Every workshop section now ships a Session Replay page and a Socratic Dialogue page — and this one replays the round that decided that.
Run this round yourself
In the workshop repo, the whole loop — frame the axis, scaffold from the kit, fan out authors, QA, hand over, read the votes, bank the taste — is one command:
md flows/this-or-that.md --_question "<WHAT TO DECIDE>" --_subject "<CONTENT HELD CONSTANT ACROSS EVERY OPTION>"
Anywhere else: paste the kickoff prompt from the section guide's prompt library, or use the this-or-that agent skill — it fetches compare.html and server.py straight from this site's /kit/ files. Both the flow file and the skill are downloadable attachments with one-line install commands on the section guide.
Grounding: the vote lines, standings table, and quoted note above are verbatim from the round's archived vote record (authoring-lab/votes-round1.jsonl) and RESULTS.md; the process beats follow the lab's README and the captured flow file. Session evidence is referred to generically. Full caveats: the section guide's appendix.