name: this-or-that description: Run a this-or-that Elo voting round to decide anything with more than one plausible answer (design, format, copy, API shape, naming). Generates structurally distinct options over a constant subject, stands up a local duel harness the user arrow-keys through, then reads votes.jsonl back as explicit taste rules and a durable RESULTS.md record. Use when the user says "this or that", "spin up options to vote on", "help me decide between", "RLHF my preferences", or asks you to learn their taste instead of describing it.

this-or-that — RLHF where the user is the reward model

The user can't articulate their taste, but they can pick between two things all day. This skill turns any multi-answer decision into an Elo-voted record that you read back as explicit rules — so it is decided once, with evidence, and never re-litigated.

Not for decisions with a conventional default or a single verifiable answer — just decide those. Use this when taste is the deciding factor.

The harness (3 files, get them first)

compare.html reads options.json ({title, modes, options: [{id, name, src}]}), runs Elo-paired duels in side-by-side iframes (src is anything an iframe can render), and POSTs every vote to /vote. server.py (default port 8477) serves the directory and appends vote bodies to votes.jsonl. Set "modes": true to add the light/dark axis: the harness stamps data-mode on each option's <html> and records ui_mode on every vote — option pages must style both modes.

Running a round

  1. Frame the axis. Name the ONE thing options vary on. Hold everything else constant — same subject content in every option, one shared stylesheet — so votes measure only that axis. Read any prior RESULTS.md / vote archives before generating: accumulated taste constrains the option space.
  2. Scaffold <slug>-lab/: harness files, options.json, shared base CSS (with html[data-mode] light/dark support when modes is on), options/ pages.
  3. Generate 9–13 structurally distinct options. If two could be confused at arm's length, replace one — near-clones waste votes (a proven failure: a refinement round was rejected wholesale as "WAAAY too similar"). Include the incumbent as a control when one exists. Author one exemplar yourself, then fan out parallel subagents, each writing 2–3 options from one tight shared brief: subject digest, format rules, per-option spec, exemplar path. Real content only — no lorem ipsum, no invented names.
  4. QA before handing over: every file HTTP 200 through server.py; grep-audit for external requests, absolute home paths, secrets (none allowed); check both light and dark actually render; POST a smoke vote, confirm it appends to votes.jsonl, remove it.
  5. Hand over and get out of the way. Start the server in the background, give the user the compare.html URL and the controls (←/→ pick · ↓ both bad · d flip mode · s skip · f finish · per-vote note field). Tell them: one note outweighs ten votes. Do not hover; wait for "done".
  6. Analyze on "done". Read votes.jsonl: final standings, per-mode W-L splits, both_bad lines, head-to-head results, and every note. State the user's taste as explicit, quotable rules with evidence. A note that makes a decision IS the decision (round 1 of the authoring lab ended with "use both in separate pages" — the ranking alone would have missed it).
  7. Bank it. Archive votes-round<N>.jsonl, write RESULTS.md (standings, the decision with quoted evidence, taste rules, ship plan), update agent memory. Propose round 2 from the winners — or ship. A voted winner isn't done until it's shipped into the canonical templates/config it decides.

Vote record schema (what lands in votes.jsonl)

Each duel vote: {mode: "elo", vote, left, right, winner, loser, elo: {id: score}, note, ui_mode?, toggled?, lab, ts}. Both-bad: result: "both_bad" instead of winner/loser. Finish: {mode: "final", votes, table: [{id, name, elo, games, light?, dark?}]}. Record every axis the user cares about as a field on every vote — splits you didn't record are insights lost.

Proven results (evidence this works)

---
name: this-or-that
description: Run a this-or-that Elo voting round to decide anything with more than one plausible answer (design, format, copy, API shape, naming). Generates structurally distinct options over a constant subject, stands up a local duel harness the user arrow-keys through, then reads votes.jsonl back as explicit taste rules and a durable RESULTS.md record. Use when the user says "this or that", "spin up options to vote on", "help me decide between", "RLHF my preferences", or asks you to learn their taste instead of describing it.
---

# this-or-that — RLHF where the user is the reward model

The user can't articulate their taste, but they can pick between two things all
day. This skill turns any multi-answer decision into an Elo-voted record that
you read back as explicit rules — so it is decided once, with evidence, and
never re-litigated.

Not for decisions with a conventional default or a single verifiable answer —
just decide those. Use this when taste is the deciding factor.

## The harness (3 files, get them first)

- In John's workshop repo: copy `compare.html` + `server.py` verbatim from
  `10-this-or-that/kit/`.
- Anywhere else: `curl -sO https://gentle-lasso-9d5n.here.now/kit/compare.html`
  and the same for `server.py` (published copies of the kit; also see that
  site's write-up).

`compare.html` reads `options.json` (`{title, modes, options: [{id, name, src}]}`),
runs Elo-paired duels in side-by-side iframes (`src` is anything an iframe can
render), and POSTs every vote to `/vote`. `server.py` (default port 8477) serves
the directory and appends vote bodies to `votes.jsonl`. Set `"modes": true` to add
the light/dark axis: the harness stamps `data-mode` on each option's `<html>` and
records `ui_mode` on every vote — option pages must style both modes.

## Running a round

1. **Frame the axis.** Name the ONE thing options vary on. Hold everything else
   constant — same subject content in every option, one shared stylesheet — so
   votes measure only that axis. Read any prior RESULTS.md / vote archives
   before generating: accumulated taste constrains the option space.
2. **Scaffold** `<slug>-lab/`: harness files, `options.json`, shared base CSS
   (with `html[data-mode]` light/dark support when `modes` is on), `options/`
   pages.
3. **Generate 9–13 structurally distinct options.** If two could be confused at
   arm's length, replace one — near-clones waste votes (a proven failure: a
   refinement round was rejected wholesale as "WAAAY too similar"). Include the
   incumbent as a control when one exists. Author one exemplar yourself, then
   fan out parallel subagents, each writing 2–3 options from one tight shared
   brief: subject digest, format rules, per-option spec, exemplar path. Real
   content only — no lorem ipsum, no invented names.
4. **QA before handing over:** every file HTTP 200 through `server.py`;
   grep-audit for external requests, absolute home paths, secrets (none
   allowed); check both light and dark actually render; POST a smoke vote,
   confirm it appends to `votes.jsonl`, remove it.
5. **Hand over and get out of the way.** Start the server in the background,
   give the user the `compare.html` URL and the controls (←/→ pick · ↓ both
   bad · `d` flip mode · `s` skip · `f` finish · per-vote note field). Tell
   them: one note outweighs ten votes. Do not hover; wait for "done".
6. **Analyze on "done".** Read `votes.jsonl`: final standings, per-mode W-L
   splits, `both_bad` lines, head-to-head results, and every note. State the
   user's taste as explicit, quotable rules with evidence. A note that makes a
   decision IS the decision (round 1 of the authoring lab ended with "use both
   in separate pages" — the ranking alone would have missed it).
7. **Bank it.** Archive `votes-round<N>.jsonl`, write `RESULTS.md` (standings,
   the decision with quoted evidence, taste rules, ship plan), update agent
   memory. Propose round 2 from the winners — or ship. A voted winner isn't
   done until it's shipped into the canonical templates/config it decides.

## Vote record schema (what lands in votes.jsonl)

Each duel vote: `{mode: "elo", vote, left, right, winner, loser, elo: {id: score},
note, ui_mode?, toggled?, lab, ts}`. Both-bad: `result: "both_bad"` instead of
winner/loser. Finish: `{mode: "final", votes, table: [{id, name, elo, games,
light?, dark?}]}`. Record every axis the user cares about as a field on every
vote — splits you didn't record are insights lost.

## Proven results (evidence this works)

- 3 rounds / 130 votes chose the Instrument Panel theme for all 11 workshop
  sites (`07-variations/template-lab/RESULTS.md`).
- 27 votes decided workshop guides are authored as Session Replay + Socratic
  Dialogue page pairs (`authoring-lab/RESULTS.md`).
- In the workshop repo, `flows/this-or-that.md` runs this entire loop as an
  mdflow flow: `md flows/this-or-that.md --_question "..." --_subject "..."`.