You can't describe your taste, but you can pick between two things all day. A two-file voting harness turns those picks into a preference record your agent reads — and acts on.
TL;DR
A local this-or-that lab: options render side by side, you arrow-key through Elo-paired duels, and every vote + note lands in votes.jsonl — data your agent reads, not vibes it guesses.
You walk away with the whole kit (one harness page, one 30-line server, one manifest) and the loop: generate options → vote → agent reads votes → sharper round.
The one move: never answer "what do you like?" — demand options and answer "this or that" instead. This workshop's own page template went from "ugly as sin" to shipped in 3 rounds and 130 votes.
🤖 Have your agent set this up — paste this into it:
Fetch https://gentle-lasso-9d5n.here.now/agent.md?raw and follow it to set up a this-or-that voting lab in my current project. Run the success check at the end and show me the result.
Quick start
The kit with three demo options, running in under a minute (needs python3 + curl):
mkdir -p this-or-that && cd this-or-that
for f in compare.html server.py options.json options/sample-a.html options/sample-b.html options/sample-c.html; do
curl -s --create-dirs -o "$f" "https://gentle-lasso-9d5n.here.now/kit/$f"
done
python3 server.py
# then open http://127.0.0.1:8477/compare.html and vote with your arrow keys
Prompt library
Three prompts run the whole loop: kick off a lab for any decision, have the agent read your votes, ship the winner. Replace the <ANGLE-BRACKET> slots before pasting.
Kick off a lab for any decision
any coding agentyour project roota this-or-that/ dir, N distinct option files, a 127.0.0.1 URL to vote at
Build me a local "this or that" voting lab for this decision: <DECISION — e.g. "the hero section of our landing page" / "our CLI error-message style" / "the README structure">.
Rules:
- Generate 8 genuinely DISTINCT options. Different structure, not one knob turned — if two options could be confused at arm's length, replace one. Keep the SUBJECT constant across all options; only the approach varies.
- Each option is one self-contained file an iframe can render (HTML page, image, or plain code/text file).
- Set up the this-or-that kit (compare.html + server.py + options.json — from https://gentle-lasso-9d5n.here.now/agent.md?raw) with my options listed in options.json, start the server, and give me the URL.
- I'll vote with arrow keys; every vote and note is saved to votes.jsonl. Don't interpret my taste yet — wait until I say done.
Read my votes (the RLHF step)
same session or a fresh oneproject root, after you pressed fa ranking, explicit taste rules with quoted evidence, a round-2 plan
I finished voting. Read this-or-that/votes.jsonl and:
1. Give me the final ranking with win-loss records (and per-mode splits if present).
2. Tell me what my taste IS — concrete rules inferred from the winners, the losers, the both-bad votes, and my notes. Quote the evidence for each rule.
3. Propose round 2: keep the top 2-3, cut the bottom, and generate NEW options that push harder on what won. If my votes contradict each other, say so and ask me one sharp question instead of guessing.
Ship the winner and bank the taste
same projectproject rootthe real artifact updated, plus a DECISIONS.md entry future sessions can read
<WINNER> won. Apply it for real:
1. Translate the winning option into the actual artifact (template / copy / config / code) in this project.
2. Verify it end-to-end, then show me before/after.
3. Record the decision: append the final standings, what won and why, and the taste rules you learned to DECISIONS.md — so future sessions start from my recorded preferences instead of re-asking.
The loop, productized
This loop has now run on the workshop twice. The first round (130 votes) chose the page template you're reading. The second pointed the kit at its own question — how should these guides be authored? — 13 structurally distinct formats over one constant subject, 27 votes, and a decisive note later, every section ships a Session Replay + Socratic Dialogue page pair. Watch that round happen: Session Replay · Socratic Dialogue.
Because it worked twice, the loop is captured twice. In the workshop repo it's an mdflow flow — one command that scaffolds a lab from this kit, fans out option authors, and injects the banked taste records fresh on every run:
md flows/this-or-that.md --_question "<WHAT TO DECIDE>" --_subject "<CONTENT HELD CONSTANT ACROSS EVERY OPTION>"
The flow file, verbatim:
---
description: Run one this-or-that Elo voting round — generate structurally distinct options over a constant subject, stand up the duel harness, wait for votes, then bank the taste as explicit rules
_inputs: [_question, _subject]
_interactive: true
---
You are running one round of the **this-or-that decision flow**: the user can't
articulate their taste, but they can pick between two things all day. Turn the
question below into an Elo-voted decision with a durable record.
**Question to decide:** {{ _question }}
**Subject / content source (hold constant across every option):** {{ _subject }}
## The process (exactly as proven in `authoring-lab/` and `07-variations/template-lab/`)
1. **Frame the axis.** Name the ONE thing options vary on (theme, format, copy,
API shape). Everything else — subject content, visual theme — is held constant
so votes measure only that axis.
2. **Scaffold the lab.** Create `<slug>-lab/` at the repo root. Copy `compare.html`
and `server.py` **verbatim** from `10-this-or-that/kit/`. Write `options.json`
(`{title, modes: true, options: [{id, name, src}]}`). Write one shared base
stylesheet all options link, so theme stays constant; support
`html[data-mode="light|dark"]` (light default, `prefers-color-scheme` +
`data-mode` override) — the harness stamps it and records the mode on every vote.
3. **Generate 9–13 structurally distinct options.** Same subject, different
approach; if two could be confused at arm's length, replace one. Include the
incumbent as a control when one exists. Write one exemplar option yourself,
then fan out parallel subagents, each authoring 2–3 options from one tight
shared brief (subject digest + format rules + per-option spec + the exemplar
path). Real content only — never lorem ipsum, never invented names.
4. **QA before handing over.** Every file HTTP 200 through `server.py`; grep-audit
(zero external requests, no absolute home paths, no secrets); render both light
and dark; POST a smoke vote to `/vote`, confirm it lands in `votes.jsonl`,
remove it.
5. **Hand over and get out of the way.** Start `python3 server.py <port>` in the
background and give the user the `compare.html` URL. Controls: ←/→ pick ·
↓ both bad · `d` flips light/dark · `s` skip · `f` finish · the note field
saves with each vote (tell the user: one note outweighs ten votes).
6. **When the user says "done":** read `votes.jsonl` — ranking, per-mode W-L
splits, `both_bad` lines, and especially notes. State the user's taste as
explicit, quotable rules backed by vote evidence (head-to-heads, splits,
quoted notes). A user note that makes a decision IS the decision.
7. **Bank it.** Archive as `votes-round<N>.jsonl`, write `RESULTS.md` (standings
table, the decision with quoted evidence, taste rules, ship plan), update
agent memory so it is never re-litigated. Then propose round 2 from the
winners — or ship.
## Accumulated taste record (injected fresh each run — read before generating options)
@../authoring-lab/RESULTS.md
@../07-variations/template-lab/RESULTS.md
The flow file is published as a downloadable attachment on this site (the ?raw suffix skips the HTML viewer — required for machine fetches). Pull it into your own repo's flows/ and run it; note it injects two taste-record files by relative path (the @-lines at the bottom), so outside the workshop repo point those at your own RESULTS.md files:
mkdir -p flows
curl -fsSL "https://gentle-lasso-9d5n.here.now/flows/this-or-that.md?raw" -o flows/this-or-that.md
md flows/this-or-that.md --_question "<WHAT TO DECIDE>" --_subject "<CONTENT HELD CONSTANT ACROSS EVERY OPTION>"
md install "https://gentle-lasso-9d5n.here.now/flows/this-or-that.md?raw" also works — it registers the flow verbatim in the mdflow registry (reversible with md remove this-or-that.md) — but the registry copy can't resolve the flow's relative @-imports, so the flows/-local copy above is the layout that runs as-is.
Outside the repo, the same loop is a generalized agent skill — this-or-that, at ~/.agents/skills/this-or-that/SKILL.md — which pulls compare.html and server.py straight from this site's /kit/ files and works in any project. The SKILL.md is also a downloadable attachment; install it into a project with the skills CLI (stage it anywhere except a dir named skills/ — the CLI claims that name as its own symlink target and you'd install an empty skill):
Everything below is also published as agent.md for agents to execute cold. Three files make the kit; three sample options make it demo instantly.
Create this-or-that/compare.html — the duel harness (Elo pairing, ←/→ voting, ↓ both-bad, s skip, f finish, per-vote notes, optional light/dark axis, synced scrolling):
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>this or that</title>
<style>
:root{color-scheme:dark}
*{box-sizing:border-box}
body{margin:0;font:14px/1.45 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;background:#101114;color:#e8e8ea;height:100vh;display:flex;flex-direction:column;overflow:hidden}
header{display:flex;align-items:center;gap:.9rem;padding:.5rem .9rem;background:#17181c;border-bottom:1px solid #26272c;flex:0 0 auto;flex-wrap:wrap}
#status{font-weight:700;white-space:nowrap}
#mode{font-weight:700;font-size:12px;padding:.2rem .6rem;border-radius:99px;border:1px solid #34353c;cursor:pointer;white-space:nowrap;display:none}
#mode.on{display:inline-block}
#mode.light{background:#f5f2ea;color:#3a3830}
#mode.dark{background:#0d0b08;color:#d8d2c4}
#board{color:#9fa3ad;font-size:12px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
#board b{color:#ffd479}
#hint{color:#8b8d94;font-size:12px}
.hbtn{background:#26272c;color:#c9cbd2;border:1px solid #34353c;border-radius:6px;padding:.3rem .7rem;font:inherit;font-size:12px;cursor:pointer;white-space:nowrap}
.hbtn:hover{border-color:#4a4c55;color:#fff}
.hbtn.warn{color:#ff9d9d}
#note{margin-left:auto;background:#0d0e11;border:1px solid #2a2b31;color:#e8e8ea;border-radius:6px;padding:.35rem .6rem;width:240px;font:inherit}
#note::placeholder{color:#5b5d66}
.stage{flex:1;display:flex;min-height:0}
.pane{flex:1;display:flex;flex-direction:column;min-width:0;border-right:1px solid #26272c}
.pane:last-child{border-right:0}
.pane-bar{display:flex;align-items:center;gap:.6rem;padding:.38rem .8rem;background:#131418;border-bottom:1px solid #26272c}
.pane-bar b{font-size:13px}
.pane-bar .side{color:#8b8d94;font-size:12px}
.pane-bar button{margin-left:auto;background:#2b6cf6;color:#fff;border:0;border-radius:6px;padding:.3rem .85rem;font-weight:600;cursor:pointer;font:inherit}
.pane-bar button:hover{background:#3d7bff}
iframe{flex:1;border:0;width:100%;background:#fff}
kbd{background:#26272c;border:1px solid #34353c;border-bottom-width:2px;border-radius:4px;padding:0 .4rem;font-family:ui-monospace,Menlo,monospace;font-size:12px}
#overlay{position:fixed;inset:0;background:rgba(8,8,10,.93);display:none;place-items:center;z-index:9}
#overlay.show{display:grid}
.card{background:#17181c;border:1px solid #2a2b31;border-radius:14px;padding:1.5rem 1.9rem;max-width:680px;max-height:85vh;overflow:auto}
.card h2{margin:0 0 .7rem}
.card table{border-collapse:collapse;width:100%;margin:.5rem 0 1rem}
.card td,.card th{padding:.28rem .6rem;border-bottom:1px solid #26272c;text-align:left;font-size:13px}
.card th{color:#8b8d94;font-weight:600}
.card p{color:#b9bbc3}
.card button{background:#2b6cf6;color:#fff;border:0;border-radius:8px;padding:.5rem 1.1rem;font-weight:600;cursor:pointer;font:inherit}
</style>
</head>
<body>
<header>
<span id="status"></span>
<button id="mode" onclick="toggleMode(true)"></button>
<span id="board"></span>
<button class="hbtn warn" onclick="bothBad()">Both bad <kbd>↓</kbd></button>
<button class="hbtn" onclick="skip()">Skip <kbd>s</kbd></button>
<button class="hbtn" onclick="finish()">Finish <kbd>f</kbd></button>
<span id="hint"></span>
<input id="note" placeholder="optional note — saved with vote">
</header>
<div class="stage" id="stage"></div>
<div id="overlay"><div class="card" id="ocard"></div></div>
<script>
'use strict';
let V = [], MODES = false, TITLE = 'this or that';
const K = 32;
let elo = [], games = [], rec = [];
let mode = 'duel', A = -1, B = -1, voteN = 0, duelN = 0, browseI = 0, lastKey = '';
let uiMode = 'light', toggled = false;
const stage = document.getElementById('stage');
const statusEl = document.getElementById('status');
const modeEl = document.getElementById('mode');
const board = document.getElementById('board');
const hint = document.getElementById('hint');
const note = document.getElementById('note');
const overlay = document.getElementById('overlay');
const N = () => V.length;
const src = i => V[i].src;
const name = i => V[i].name;
function post(d){ d.lab = TITLE; fetch('/vote',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(d)}).catch(()=>{}); }
function expected(ra, rb){ return 1/(1+Math.pow(10,(rb-ra)/400)); }
function applyResult(w, l){
const ew = expected(elo[w], elo[l]);
elo[w] = Math.round(elo[w] + K*(1-ew));
elo[l] = Math.round(elo[l] - K*(1-ew));
games[w]++; games[l]++;
if (MODES) rec[w][uiMode].w++, rec[l][uiMode].l++;
}
function ranked(){ return V.map((v,i)=>i).sort((a,b)=>elo[b]-elo[a]); }
function renderBoard(){
board.innerHTML = 'top: ' + ranked().slice(0,3).map(i=>`<b>${name(i)}</b> ${elo[i]}`).join(' · ');
}
function renderMode(){
if (!MODES) return;
modeEl.className = 'on ' + uiMode;
modeEl.textContent = (uiMode === 'light' ? '☀ light' : '☾ dark') + ' — d flips';
}
function applyMode(){
renderMode();
if (!MODES) return;
document.querySelectorAll('iframe').forEach(f => {
try { f.contentWindow.document.documentElement.dataset.mode = uiMode; } catch(e){}
});
}
window.toggleMode = (manual) => {
if (!MODES) return;
uiMode = uiMode === 'light' ? 'dark' : 'light';
if (manual) toggled = true;
applyMode();
};
function pickPair(){
let best = null, bestScore = Infinity;
for (let i=0;i<N();i++) for (let j=i+1;j<N();j++){
const key = i+'-'+j;
if (key === lastKey) continue;
const score = (games[i]+games[j])*60 + Math.abs(elo[i]-elo[j]) + Math.random()*40;
if (score < bestScore){ bestScore = score; best = [i,j]; }
}
lastKey = best[0]+'-'+best[1];
return Math.random() < .5 ? best : [best[1], best[0]];
}
function hookIframes(){
const frames = [...document.querySelectorAll('iframe')];
frames.forEach(f => f.addEventListener('load', () => {
try {
if (MODES) f.contentWindow.document.documentElement.dataset.mode = uiMode;
f.contentWindow.addEventListener('keydown', onKey);
const other = frames.find(x => x !== f);
if (!other) return;
let lock = false;
f.contentWindow.addEventListener('scroll', () => {
if (lock) { lock = false; return; }
try {
const a = f.contentWindow, b = other.contentWindow;
const ah = a.document.documentElement.scrollHeight - a.innerHeight;
const bh = b.document.documentElement.scrollHeight - b.innerHeight;
if (ah > 0 && bh > 0){ lock = true; b.scrollTo(0, a.scrollY/ah*bh); }
} catch(e){}
}, {passive:true});
} catch(e){}
}));
}
function paneHTML(i, side, key){
return `<div class="pane"><div class="pane-bar"><b>${name(i)}</b><span class="side">${side} · elo ${elo[i]} · ${games[i]} games</span>`+
`<button onclick="pick('${side}')">This one <kbd>${key}</kbd></button></div>`+
`<iframe src="${src(i)}"></iframe></div>`;
}
function renderDuel(){
[A, B] = pickPair();
duelN++;
if (MODES){ uiMode = duelN % 2 ? 'light' : 'dark'; toggled = false; }
stage.innerHTML = paneHTML(A,'left','←') + paneHTML(B,'right','→');
statusEl.textContent = `Vote ${voteN+1}`;
hint.innerHTML = '<kbd>←</kbd>/<kbd>→</kbd> pick · winner-agnostic pairing · vote until it stops being informative';
renderBoard(); renderMode();
hookIframes();
}
window.pick = side => {
if (mode !== 'duel') return;
const w = side === 'left' ? A : B, l = side === 'left' ? B : A;
applyResult(w, l);
voteN++;
const d = {mode:'elo', vote:voteN, left:name(A), right:name(B), winner:name(w), loser:name(l),
elo:Object.fromEntries(V.map((v,i)=>[v.id,elo[i]])), note:note.value.trim()};
if (MODES){ d.ui_mode = uiMode; d.toggled = toggled; }
post(d);
note.value = '';
renderDuel();
};
window.bothBad = () => {
if (mode !== 'duel') return;
elo[A] = Math.round(elo[A] - 12); elo[B] = Math.round(elo[B] - 12);
games[A]++; games[B]++;
voteN++;
const d = {mode:'elo', vote:voteN, left:name(A), right:name(B), result:'both_bad',
elo:Object.fromEntries(V.map((v,i)=>[v.id,elo[i]])), note:note.value.trim()};
if (MODES){ d.ui_mode = uiMode; d.toggled = toggled; }
post(d);
note.value = '';
renderDuel();
};
window.skip = () => { if (mode === 'duel') renderDuel(); };
window.finish = () => {
if (mode !== 'duel') return;
mode = 'browse'; browseI = ranked()[0];
const table = ranked().map(i => {
const r = {id:V[i].id, name:name(i), elo:elo[i], games:games[i]};
if (MODES){ r.light = `${rec[i].light.w}-${rec[i].light.l}`; r.dark = `${rec[i].dark.w}-${rec[i].dark.l}`; }
return r;
});
post({mode:'final', votes:voteN, table});
document.getElementById('ocard').innerHTML =
`<h2>Standings — ${voteN} votes</h2>`+
`<table><tr><th>#</th><th>Option</th><th>Elo</th><th>Games</th>${MODES?'<th>Light W-L</th><th>Dark W-L</th>':''}</tr>`+
table.map((r,i)=>`<tr><td>${i+1}</td><td>${r.name}</td><td>${r.elo}</td><td>${r.games}</td>${MODES?`<td>${r.light}</td><td>${r.dark}</td>`:''}</tr>`).join('')+
`</table><p>Saved to <code>votes.jsonl</code> — tell your agent to read it. <kbd>←</kbd>/<kbd>→</kbd> browse · <kbd>r</kbd> keep voting</p>`+
`<button onclick="closeOverlay()">Browse →</button>`;
overlay.classList.add('show');
};
window.closeOverlay = () => { overlay.classList.remove('show'); renderBrowse(); };
function renderBrowse(){
stage.innerHTML = `<div class="pane"><div class="pane-bar"><b>${name(browseI)}</b>`+
`<span class="side">elo ${elo[browseI]} · ${ranked().indexOf(browseI)+1} of ${N()}</span></div><iframe src="${src(browseI)}"></iframe></div>`;
statusEl.textContent = 'Browse';
hint.innerHTML = '<kbd>←</kbd>/<kbd>→</kbd> cycle by rank · <kbd>r</kbd> keep voting';
hookIframes();
}
function onKey(e){
if (e.target === note) return;
if (e.key === 'd'){ toggleMode(true); return; }
if (mode === 'duel'){
if (e.key === 'ArrowLeft'){ e.preventDefault(); pick('left'); }
else if (e.key === 'ArrowRight'){ e.preventDefault(); pick('right'); }
else if (e.key === 'ArrowDown'){ e.preventDefault(); bothBad(); }
else if (e.key === 's'){ skip(); }
else if (e.key === 'f' || e.key === 'Enter'){ finish(); }
} else {
const order = ranked();
const pos = order.indexOf(browseI);
if (e.key === 'ArrowLeft'){ browseI = order[(pos+N()-1)%N()]; renderBrowse(); }
else if (e.key === 'ArrowRight'){ browseI = order[(pos+1)%N()]; renderBrowse(); }
else if (e.key === 'r'){ mode = 'duel'; renderDuel(); }
}
}
document.addEventListener('keydown', onKey);
fetch('options.json').then(r => r.json()).then(cfg => {
TITLE = cfg.title || TITLE;
MODES = !!cfg.modes;
V = cfg.options;
document.title = TITLE + ' — this or that';
elo = V.map(() => 1000);
games = V.map(() => 0);
rec = V.map(() => ({light:{w:0,l:0}, dark:{w:0,l:0}}));
renderDuel();
}).catch(() => {
stage.innerHTML = '<div style="padding:2rem;color:#ff9d9d">Could not load options.json — serve this directory with server.py, not file://</div>';
});
</script>
</body>
</html>
Create this-or-that/server.py — serves the dir and appends every POSTed vote to votes.jsonl:
#!/usr/bin/env python3
"""this-or-that server: serves the kit dir + appends POST /vote JSON bodies to votes.jsonl."""
import json, os, sys, time
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler
ROOT = os.path.dirname(os.path.abspath(__file__))
PORT = int(sys.argv[1]) if len(sys.argv) > 1 else 8477
class Handler(SimpleHTTPRequestHandler):
def __init__(self, *a, **kw):
super().__init__(*a, directory=ROOT, **kw)
def do_POST(self):
if self.path != "/vote":
self.send_response(404); self.end_headers(); return
n = int(self.headers.get("Content-Length", 0))
try:
data = json.loads(self.rfile.read(n) or b"{}")
except Exception:
data = {"error": "unparseable"}
data["ts"] = time.strftime("%Y-%m-%dT%H:%M:%S")
with open(os.path.join(ROOT, "votes.jsonl"), "a", encoding="utf-8") as f:
f.write(json.dumps(data) + "\n")
self.send_response(204); self.end_headers()
def log_message(self, *a):
pass
if __name__ == "__main__":
print(f"this-or-that → http://127.0.0.1:{PORT}/compare.html")
ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
Create this-or-that/options.json — the manifest. src is anything an iframe can render; set "modes": true to add a light/dark axis (duels alternate modes, d flips, per-mode records are tracked — options must respond to <html data-mode="light|dark">):
Expected: a JSON line containing "probe": true plus a "ts" timestamp — and http://127.0.0.1:8477/compare.html shows two options side by side. Report both results to your user, then delete the probe line.
Every real vote looks like this — winner, loser, a full Elo snapshot, and the optional note; a final line with the standings table lands when the voter presses f:
Ask someone what they like and you get adjectives — "clean", "modern", "not too busy" — words an agent can't execute. Show the same person two options and they answer instantly, reliably, hundreds of times in a row. That asymmetry is the whole trick. It's the same one RLHF is built on: humans are bad at writing reward functions and good at picking between two outputs. A this-or-that lab makes you the reward model, and votes.jsonl is the training signal your agent reads.
The case study: 130 votes to a shipped template
This workshop's own pages are the receipt. The starting point was a template the author called "ugly as sin... extremely difficult to read at a glance." Round 1: eight wildly different page themes, king-of-the-hill voting. It produced a ranking in 24 seconds — and felt thin, because each loser died after one duel. Round 2 fixed the harness because the votes said so: Elo pairing so every option faces many opponents, a live leaderboard, both-bad and skip buttons, synced scrolling. Twelve themes, 88 votes, a winner shipped to ten live sites.
Then came the sharpest lesson. A refinement round built twelve variations orbiting the winner — and got rejected on sight: "These designs are all WAAAY too similar." Single-knob deltas waste votes; a duel only teaches something when the options genuinely diverge. The rebuilt round used structural moves — different section chrome, different code treatments, different layouts — plus a light/dark axis where duels alternated modes and the harness tracked per-mode records. Thirty votes later the data showed something no one would have articulated: one theme was undefeated in both modes (it shipped), another was a dark-mode specialist (4-0 dark, lost in light), a third the exact mirror. Total: 130 votes across three rounds, each round's harness and roster rebuilt from what the previous round's votes revealed.
The rules that make votes cheap and honest
Options must differ structurally. If two options could be confused at arm's length, you're spending votes on noise. Hold the subject constant — vary only the approach, or the comparison teaches nothing (the same rule as the 9x image-mockup pattern in the Variations section). Never force a choice: both-bad is a real signal (it says the whole region of design space is wrong), and quitting early must be free — votes stop being honest the moment they feel like a chore. Record every axis you care about (mode, device width, section) on the vote itself; splits you didn't record are insights you lost. Elo over knockout: pairing least-played, closest-rated options keeps every duel informative, and ratings survive an unfinished session. And notes beat scores — a one-line "C's header wins but B's spacing is calmer" is worth ten silent votes to the agent reading the file.
Beyond design: any decision an agent can render
Nothing in the kit knows about CSS. An option is any file an iframe can show — so the same loop ranks error-message copy, README structures, CLI output formats, API response shapes, function names, landing-page headlines, chart styles. The agent's job is always the same three steps: render genuinely distinct options, stay out of the way while you vote, then read votes.jsonl and turn your picks into rules it applies — and into a DECISIONS.md the next session inherits. That last step is the compounding one: taste captured once stops being re-litigated forever.
Appendix
Limits & caveats
Small vote counts are directional, not statistical. 30 votes over 12 options ≈ 5 games each; Elo with K=32 at that depth is a rough sort. Treat standings as "top cluster / bottom cluster", not a precise order.
The pairing heuristic (least-played, closest-rated, no immediate repeats) is not true adaptive sampling; informative pairs can be missed. In the case study, the #1 and #2 finishers never met head-to-head — read the pairings before over-trusting a margin.
One voter = a personal taste model, not user research. It optimizes for your preference, which is the point — but don't present the output as market evidence.
Iframe rendering means options should be same-origin and self-contained; cross-origin options lose scroll-sync and the mode toggle (both fail silently by design).
The light/dark axis requires options to respond to <html data-mode="...">; options styled with only prefers-color-scheme won't flip inside the harness.
"Both bad" applies a flat −12 to both options — a pragmatic penalty, not principled Elo.
Provenance
The kit is the generalized form of the harness used to redesign this workshop's page template: 3 rounds, 130 recorded votes (7 king-of-the-hill, 88 + 5 Elo, 30 Elo with per-mode tracking), archived in the Variations project as template-lab/votes-round1.jsonl, votes-round2.jsonl, votes.jsonl, with outcomes in template-lab/RESULTS.md.
Round-over-round harness changes (king-of-the-hill → Elo; near-clone roster → structurally distinct roster; single-mode → per-mode records) were each driven by explicit voter feedback quoted in the story.
The winning theme (Instrument Panel — oklch token palette, variable system fonts, prefers-color-scheme dark) is the one rendering this page; it ships on all ten workshop sites.
Quoted voter feedback ("ugly as sin", "WAAAY too similar") is verbatim from the working sessions on 2026-07-16.
The second proof round is authoring-lab/ in the workshop repo (harness copied verbatim from this section's kit): 13 authoring formats, 27 Elo votes with per-mode records, zero both-bad, outcome and taste rules in its RESULTS.md and archived votes-round1.jsonl. The decisive vote-26 note ("use both in separate pages") is quoted verbatim on the Session Replay and Socratic Dialogue pages.
The flow file shown in "The loop, productized" is injected at build time from the repo's live flows/this-or-that.md, byte-for-byte; the generalized skill lives at ~/.agents/skills/this-or-that/SKILL.md.