Agentic Software Factory · 2026-07-17 · This or That: RLHF for Your Agent

“Why not just tell the agent what I like?”

The section as the workshop room actually runs it: skeptical questions drive the reveal — what votes cost, what stops near-clones, the note that overruled the ranking, and how to rerun the loop without re-explaining yourself.

The room asks

Why not just tell the agent what I like?

Try it — you'll produce adjectives. "Clean." "Modern." "Not too busy." An agent can't execute an adjective. But show the same person two options and they answer instantly, honestly, hundreds of times in a row. That asymmetry is what RLHF is built on: humans are bad at writing reward functions and good at picking between two outputs. The harness just makes you the reward model, and instead of a vibe the agent gets a file. Every keystroke lands like this:

{"mode": "elo", "vote": 7, "left": "Installable Skill", "right": "Session Replay",
 "winner": "Session Replay", "loser": "Installable Skill",
 "elo": { …all 13 ratings after the vote… },
 "note": "", "ui_mode": "light", "lab": "How should workshop projects be authored?"}
Twenty-seven votes sounds like homework. How long did that actually take?

About ten minutes, by the timestamps in the vote record. A duel is one keystroke; notes are optional; quitting early is free — Elo standings survive an unfinished session, which is exactly what keeps votes honest instead of dutiful. For scale: the template rendering this page took 130 votes across three rounds, and that's still cheaper than one "can you make it pop?" meeting.

What stops the options from all looking the same?

A round that got thrown out. A refinement pass once built twelve near-clones of a winning theme and was rejected on sight — "These designs are all WAAAY too similar." Single-knob deltas waste votes; a duel only teaches something when the options genuinely diverge. So the rule is structural, and it's written into the captured flow:

3. **Generate 9–13 structurally distinct options.** Same subject, different
   approach; if two could be confused at arm's length, replace one. Include the
   incumbent as a control when one exists.

The other half is holding the subject constant. In the authoring round, all 13 formats rendered the same zsh material — so every vote measured format and nothing else.

Dialogue won that round. So why am I looking at two pages?

Because of one note on duel 26 — the top two head-to-head:

{"vote": 26, "left": "Session Replay", "right": "Socratic Dialogue",
 "winner": "Socratic Dialogue", "ui_mode": "dark", "toggled": true,
 "note": "I love _both_ the \"session replay - custom zsh functions\" and
          \"dialogue - custom zsh functions\". I think we could use both in
          separate pages so that each project has a session replay and a
          dialogue page"}

The ranking said "ship Dialogue." The note said "ship both, as a pair" — and the pair is what shipped, on every section of this workshop. If the harness only recorded winners, that decision would have been lost. That's why the note field exists, and why every axis you care about (light/dark here) is recorded on the vote itself.

I pressed f. What do I actually do with votes.jsonl?

Hand it to the agent — this is the RLHF step, and it's a prompt you can lift as-is:

I finished voting. Read this-or-that/votes.jsonl and:
1. Give me the final ranking with win-loss records (and per-mode splits if present).
2. Tell me what my taste IS — concrete rules inferred from the winners, the losers, the both-bad votes, and my notes. Quote the evidence for each rule.
3. Propose round 2: keep the top 2-3, cut the bottom, and generate NEW options that push harder on what won. If my votes contradict each other, say so and ask me one sharp question instead of guessing.

Then bank the outcome: standings, the quoted evidence, and the taste rules go into a RESULTS.md and the raw record gets archived as votes-round<N>.jsonl. Future sessions start from your recorded preferences instead of re-asking. Taste captured once stops being re-litigated forever.

And next month, when the next decision comes up — do I re-explain all of this?

No. In the workshop repo the whole loop is one command; the flow injects the banked taste records fresh on every run, so round N+1 starts where round N ended:

md flows/this-or-that.md --_question "<WHAT TO DECIDE>" --_subject "<CONTENT HELD CONSTANT ACROSS EVERY OPTION>"

The flow file itself is short enough to read on one screen — and to lift:

---
description: Run one this-or-that Elo voting round — generate structurally distinct options over a constant subject, stand up the duel harness, wait for votes, then bank the taste as explicit rules
_inputs: [_question, _subject]
_interactive: true
---

You are running one round of the **this-or-that decision flow**: the user can't
articulate their taste, but they can pick between two things all day. Turn the
question below into an Elo-voted decision with a durable record.

**Question to decide:** {{ _question }}

**Subject / content source (hold constant across every option):** {{ _subject }}

## The process (exactly as proven in `authoring-lab/` and `07-variations/template-lab/`)

1. **Frame the axis.** Name the ONE thing options vary on (theme, format, copy,
   API shape). Everything else — subject content, visual theme — is held constant
   so votes measure only that axis.
2. **Scaffold the lab.** Create `<slug>-lab/` at the repo root. Copy `compare.html`
   and `server.py` **verbatim** from `10-this-or-that/kit/`. Write `options.json`
   (`{title, modes: true, options: [{id, name, src}]}`). Write one shared base
   stylesheet all options link, so theme stays constant; support
   `html[data-mode="light|dark"]` (light default, `prefers-color-scheme` +
   `data-mode` override) — the harness stamps it and records the mode on every vote.
3. **Generate 9–13 structurally distinct options.** Same subject, different
   approach; if two could be confused at arm's length, replace one. Include the
   incumbent as a control when one exists. Write one exemplar option yourself,
   then fan out parallel subagents, each authoring 2–3 options from one tight
   shared brief (subject digest + format rules + per-option spec + the exemplar
   path). Real content only — never lorem ipsum, never invented names.
4. **QA before handing over.** Every file HTTP 200 through `server.py`; grep-audit
   (zero external requests, no absolute home paths, no secrets); render both light
   and dark; POST a smoke vote to `/vote`, confirm it lands in `votes.jsonl`,
   remove it.
5. **Hand over and get out of the way.** Start `python3 server.py <port>` in the
   background and give the user the `compare.html` URL. Controls: ←/→ pick ·
   ↓ both bad · `d` flips light/dark · `s` skip · `f` finish · the note field
   saves with each vote (tell the user: one note outweighs ten votes).
6. **When the user says "done":** read `votes.jsonl` — ranking, per-mode W-L
   splits, `both_bad` lines, and especially notes. State the user's taste as
   explicit, quotable rules backed by vote evidence (head-to-heads, splits,
   quoted notes). A user note that makes a decision IS the decision.
7. **Bank it.** Archive as `votes-round<N>.jsonl`, write `RESULTS.md` (standings
   table, the decision with quoted evidence, taste rules, ship plan), update
   agent memory so it is never re-litigated. Then propose round 2 from the
   winners — or ship.

## Accumulated taste record (injected fresh each run — read before generating options)

@../authoring-lab/RESULTS.md

@../07-variations/template-lab/RESULTS.md

Outside this repo, the same loop is a generalized agent skill (this-or-that, at ~/.agents/skills/this-or-that/SKILL.md) — it pulls compare.html and server.py straight from this site's /kit/ files. Both the flow file and the skill are downloadable attachments with one-line install commands on the section guide.

Isn't one person's 27 votes just… bias?

Yes — deliberately. This is a personal taste model, not user research; don't present the standings as market evidence. The point is that your bias becomes data an agent can act on and cite back at you, instead of a mood it has to guess at. And it isn't limited to design: an option is anything an iframe can render — error copy, README structures, API shapes, function names. The authoring round voted on formats, not colors, and it ended with the page pair you're navigating right now. Want to watch that round happen turn by turn? That's the Session Replay page.

Every standing, quote, and vote excerpt above is verbatim from the two banked voting rounds — the template lab (130 votes) and the authoring lab (27 votes) — via their RESULTS.md files and archived vote records. Session evidence is referred to generically. Full caveats: the section guide's appendix.