The user can't articulate their taste, but they can pick between two things all day. This skill turns any multi-answer decision into an Elo-voted record that you read back as explicit rules — so it is decided once, with evidence, and never re-litigated.
Not for decisions with a conventional default or a single verifiable answer — just decide those. Use this when taste is the deciding factor.
compare.html + server.py verbatim from
10-this-or-that/kit/.curl -sO https://gentle-lasso-9d5n.here.now/kit/compare.html
and the same for server.py (published copies of the kit; also see that
site's write-up).compare.html reads options.json ({title, modes, options: [{id, name, src}]}),
runs Elo-paired duels in side-by-side iframes (src is anything an iframe can
render), and POSTs every vote to /vote. server.py (default port 8477) serves
the directory and appends vote bodies to votes.jsonl. Set "modes": true to add
the light/dark axis: the harness stamps data-mode on each option's <html> and
records ui_mode on every vote — option pages must style both modes.
<slug>-lab/: harness files, options.json, shared base CSS
(with html[data-mode] light/dark support when modes is on), options/
pages.server.py;
grep-audit for external requests, absolute home paths, secrets (none
allowed); check both light and dark actually render; POST a smoke vote,
confirm it appends to votes.jsonl, remove it.compare.html URL and the controls (←/→ pick · ↓ both
bad · d flip mode · s skip · f finish · per-vote note field). Tell
them: one note outweighs ten votes. Do not hover; wait for "done".votes.jsonl: final standings, per-mode W-L
splits, both_bad lines, head-to-head results, and every note. State the
user's taste as explicit, quotable rules with evidence. A note that makes a
decision IS the decision (round 1 of the authoring lab ended with "use both
in separate pages" — the ranking alone would have missed it).votes-round<N>.jsonl, write RESULTS.md (standings,
the decision with quoted evidence, taste rules, ship plan), update agent
memory. Propose round 2 from the winners — or ship. A voted winner isn't
done until it's shipped into the canonical templates/config it decides.Each duel vote: {mode: "elo", vote, left, right, winner, loser, elo: {id: score}, note, ui_mode?, toggled?, lab, ts}. Both-bad: result: "both_bad" instead of
winner/loser. Finish: {mode: "final", votes, table: [{id, name, elo, games, light?, dark?}]}. Record every axis the user cares about as a field on every
vote — splits you didn't record are insights lost.
07-variations/template-lab/RESULTS.md).authoring-lab/RESULTS.md).flows/this-or-that.md runs this entire loop as an
mdflow flow: md flows/this-or-that.md --_question "..." --_subject "...".---
name: this-or-that
description: Run a this-or-that Elo voting round to decide anything with more than one plausible answer (design, format, copy, API shape, naming). Generates structurally distinct options over a constant subject, stands up a local duel harness the user arrow-keys through, then reads votes.jsonl back as explicit taste rules and a durable RESULTS.md record. Use when the user says "this or that", "spin up options to vote on", "help me decide between", "RLHF my preferences", or asks you to learn their taste instead of describing it.
---
# this-or-that — RLHF where the user is the reward model
The user can't articulate their taste, but they can pick between two things all
day. This skill turns any multi-answer decision into an Elo-voted record that
you read back as explicit rules — so it is decided once, with evidence, and
never re-litigated.
Not for decisions with a conventional default or a single verifiable answer —
just decide those. Use this when taste is the deciding factor.
## The harness (3 files, get them first)
- In John's workshop repo: copy `compare.html` + `server.py` verbatim from
`10-this-or-that/kit/`.
- Anywhere else: `curl -sO https://gentle-lasso-9d5n.here.now/kit/compare.html`
and the same for `server.py` (published copies of the kit; also see that
site's write-up).
`compare.html` reads `options.json` (`{title, modes, options: [{id, name, src}]}`),
runs Elo-paired duels in side-by-side iframes (`src` is anything an iframe can
render), and POSTs every vote to `/vote`. `server.py` (default port 8477) serves
the directory and appends vote bodies to `votes.jsonl`. Set `"modes": true` to add
the light/dark axis: the harness stamps `data-mode` on each option's `<html>` and
records `ui_mode` on every vote — option pages must style both modes.
## Running a round
1. **Frame the axis.** Name the ONE thing options vary on. Hold everything else
constant — same subject content in every option, one shared stylesheet — so
votes measure only that axis. Read any prior RESULTS.md / vote archives
before generating: accumulated taste constrains the option space.
2. **Scaffold** `<slug>-lab/`: harness files, `options.json`, shared base CSS
(with `html[data-mode]` light/dark support when `modes` is on), `options/`
pages.
3. **Generate 9–13 structurally distinct options.** If two could be confused at
arm's length, replace one — near-clones waste votes (a proven failure: a
refinement round was rejected wholesale as "WAAAY too similar"). Include the
incumbent as a control when one exists. Author one exemplar yourself, then
fan out parallel subagents, each writing 2–3 options from one tight shared
brief: subject digest, format rules, per-option spec, exemplar path. Real
content only — no lorem ipsum, no invented names.
4. **QA before handing over:** every file HTTP 200 through `server.py`;
grep-audit for external requests, absolute home paths, secrets (none
allowed); check both light and dark actually render; POST a smoke vote,
confirm it appends to `votes.jsonl`, remove it.
5. **Hand over and get out of the way.** Start the server in the background,
give the user the `compare.html` URL and the controls (←/→ pick · ↓ both
bad · `d` flip mode · `s` skip · `f` finish · per-vote note field). Tell
them: one note outweighs ten votes. Do not hover; wait for "done".
6. **Analyze on "done".** Read `votes.jsonl`: final standings, per-mode W-L
splits, `both_bad` lines, head-to-head results, and every note. State the
user's taste as explicit, quotable rules with evidence. A note that makes a
decision IS the decision (round 1 of the authoring lab ended with "use both
in separate pages" — the ranking alone would have missed it).
7. **Bank it.** Archive `votes-round<N>.jsonl`, write `RESULTS.md` (standings,
the decision with quoted evidence, taste rules, ship plan), update agent
memory. Propose round 2 from the winners — or ship. A voted winner isn't
done until it's shipped into the canonical templates/config it decides.
## Vote record schema (what lands in votes.jsonl)
Each duel vote: `{mode: "elo", vote, left, right, winner, loser, elo: {id: score},
note, ui_mode?, toggled?, lab, ts}`. Both-bad: `result: "both_bad"` instead of
winner/loser. Finish: `{mode: "final", votes, table: [{id, name, elo, games,
light?, dark?}]}`. Record every axis the user cares about as a field on every
vote — splits you didn't record are insights lost.
## Proven results (evidence this works)
- 3 rounds / 130 votes chose the Instrument Panel theme for all 11 workshop
sites (`07-variations/template-lab/RESULTS.md`).
- 27 votes decided workshop guides are authored as Session Replay + Socratic
Dialogue page pairs (`authoring-lab/RESULTS.md`).
- In the workshop repo, `flows/this-or-that.md` runs this entire loop as an
mdflow flow: `md flows/this-or-that.md --_question "..." --_subject "..."`.