Goal Seek: Choosing the Room to Get the Verdict

Interactive sweep dashboard: every mix, every panel, and the budget curves.

The question

Name an outcome you want from a group. Simulate that group many times over many different compositions. Is there a mix of people that reliably produces the outcome, can a search find it, and does the answer survive being re-run?

Everything else in this program asks whether a simulation matches something real. This asks the inverse: whether a simulation can be steered to a named target by choosing who sits in the room. That is the shape of a real product request, and it is the shape of the claim most likely to be oversold, because a search over a stochastic simulator always returns a winner whether or not one exists.

Setup

The case is RUIZ v. NORTHSTAR MARKETS, a fictional civil trial record already written for a simulated jury exercise. It is genuinely balanced. Notice turns on whether a customer’s report of a wet floor reached an employee, nobody established where the liquid came from, and the plaintiff’s phone log supports a comparative-fault argument.

Six jurors deliberate for two rounds in seat order, so later speakers answer earlier ones, then each fills in a verdict form privately. Majority decides, with ties going to the defence because the plaintiff carries the burden.

A mix says how many of the six seats take each level on five binary axes. Four are demographic. The fifth is a belief that bears on the disputed question, included as a positive control so that “the method detects nothing” can be told apart from “demographics carry no signal”.

That gives 7⁵ = 16,807 possible mixes. Asking each juror alone is cheap enough that the whole space can be mapped without further model calls; running the discussion is not, so only twelve mixes were deliberated.

Choosing the goal was most of the work

Three candidate goals turned out to be unsteerable, and finding that out cost about 300 model calls.

• Liability is at the ceiling: 0.93, 0.99 and 0.97 across three models. A six-seat majority reaches “liable” essentially always.
• Comparative fault collapses. One model put 49 of 85 answers at 0% and 30 at 100%.
• Whether the plaintiff was careless sits at 0.06. Floor.

The goal that survived is the one the record deliberately leaves open: award the disputed $9,600 for future shoulder care, where the treating orthopedist and the defence radiologist genuinely disagree. Base rate 0.48 to 0.50 on llama3.1:8b, which is as much room as a binary outcome can have.

This is worth stating as a finding rather than a footnote. Goal seek has room only where the case is close, and a few hundred calls tell you whether it is close before you commit to anything.

Result 1: demographics move nothing, one relevant belief moves a lot

32 profiles, 24 replicates each, 768 calls on llama3.1:8b at a 0.499 base rate. The test permutes profile labels within replicate 20,000 times, which preserves whatever the random seed was doing and destroys whatever the persona was doing. Holm-corrected across the five axes.

The control fires hard, so the method can detect an effect. The four demographic axes are flat on the model with the most headroom, at a sample size that resolves an effect a third the size of the control’s.

This is the boundary from our work on documented individuals reappearing inside group discussion. There, one measured fact about a person moved individual accuracy by about a quarter of the scale while traits invented from that person’s demographics landed below a demographics-only baseline. Here, a belief about the question in front of the jury moves a group outcome 13.8 points while demographic description moves nothing measurable. A room is steerable through what its members already think about the question, not through who they are.

It also leaves four axes known to be inert, which is a free measurement of how often a search finds something in noise.

Result 2: the surface over-promises about threefold

The map of the mix space is built from 32 per-profile rates, each estimated from 48 draws, so each carries a standard error near 0.07. Taking the extreme of thousands of mixes built on 32 noisy numbers selects for the profiles that got lucky. The naive map spans 0.105 to 0.714, and its best mix loads on three axes already shown to be inert.

Split-half test: build the map on half of each profile’s draws, take its best mix, score that mix on the other half. 24 splits, 2,500 candidate mixes.

Both estimators put the one axis that matters at zero seats, so the search finds the right lever. What it gets wrong is the size of the prize, by roughly a factor of three. Shrinking to an additive model halves the over-promise and keeps a larger share of what it claims.

The realised lift is real: about +0.16 over a representative panel, from seating six jurors who already believe an old injury can be aggravated.

Result 3: the over-promise is a property of the search budget

Simulated searches against the fitted surface, 300 independent searches per cell. “Over-promise” is the reported rate at the winning mix minus its true rate.

Two regularities, both of which should generalise past this case. The over-promise grows with the number of mixes searched at a fixed budget, which is the ordinary multiple-comparisons effect and means a wider search is a more dishonest one unless the budget grows with it. And it shrinks with panels per mix without ever reaching zero.

Fitting a response surface instead of taking the raw maximum cuts the over-promise by roughly a factor of seven at a realistic budget, and finds a mix with more than twice the realised lift (+0.103 against +0.045 at 100 mixes and 10 panels each), because the penalty pulls a lucky mix back toward its neighbours.

Result 4: does the cheap arm predict the expensive one?

Twelve mixes, eight deliberating panels each, 1,728 model calls.

The free per-juror map and the deliberating panels agree at Pearson r = +0.624 (95% interval +0.081 to +0.892, p = 0.030) and Spearman ρ = +0.575. Twelve mixes is not many, and the interval says so.

Deliberation does not wash composition out. Across the same twelve mixes the spread is 0.314 before discussion and 0.375 after, so talking slightly widens the gap between compositions rather than closing it. It does shift the whole panel upward by +0.221: a room that discusses this case awards the disputed care more often than the same people would alone.

Read against the raw per-profile map instead, the same comparison appears to show deliberation compressing the range from 0.609 to 0.375. That apparent compression is entirely an artefact of the inflated map, and it is the clearest demonstration we have that the shrunk estimator is the one to reason from.

Result 5: re-running the winner

The searched winner, the mix the free map liked best, and a representative panel, each re-run on 20 panels the search never saw.

The mix the search crowned at 0.750 returns 0.600 on fresh panels, a drop of 0.150. Three of the twelve mixes had been tied at the top, which is what an eight-panel budget buys.

Against a representative panel the search appeared to promise +0.375. What survives is +0.100, with a 95% interval of −0.207 to +0.407 (Fisher exact p = 0.75). So the point estimate keeps about a quarter of the promised advantage, and at 20 panels per mix the advantage is not distinguishable from nothing at all. The pilot is too small to say the winning mix beats a jury drawn at random, and saying otherwise from the searched number alone would have been wrong by 0.275.

One comparison stands out. The free per-juror map predicted 0.498 for the winning mix and the truth came back 0.600, an error of 0.102. The deliberating search, which cost 1,728 model calls, predicted 0.750 for the same mix, an error of 0.150. On this run the cheap arm was the better predictor of the expensive arm’s own held-out behaviour, because it was never given the chance to overfit to eight panels.

What would make this a paper

• More than one case, one outcome, and one panel size.
• More models. Of the three tried, one sat at a 0.06 floor and told us nothing.
• Demographic axes that are richer than four binary contrasts. A null on these is not a null on demographics in general.
• A real-world anchor. Everything here is a simulation validated against another simulation, so every number is about internal consistency and none of it says what a human jury would do.
• Some way to handle the assumption that axes are independent when a panel is built, which is what a marginal audience specification implies and is not true of real juries.

A second case, because the first could not answer its own question

Added 2026-09-02.

The trial case has a flaw we only saw by reading the transcripts afterwards. Nobody knows whether age or education should change a verdict about a wet floor, so a flat demographic result was equally consistent with “the simulation ignores who people are” and “this was never a demographic question.” Two further defects were ours: 77% of what jurors said re-argued a question the judge had already decided, and a juror’s own words predicted that juror’s own ballot only weakly.

So we built a second case where the right answer is known in advance. A citizen board votes on a light rail extension paid for by a sales tax. That is a services-versus-taxes question, which the American National Election Studies measures directly, so we know how far each demographic really moves real people: 18.9 points for age, 14.7 for income, 8.0 for sex. A flat result is now a failure of the simulation rather than a fact about the case.

The simulation reproduces none of it

llama3.1:8b, 32 member profiles, 24 draws each, 768 calls, all parsed. Board recommendation rate 0.521, about as contested as a yes-or-no question gets.

Every real gap falls outside the range the simulation produces. Two point the wrong way. The last row is the same measurement on a member told what they believe rather than who they are, and it nearly decides the outcome on its own.

One caveat on the target: the survey gap is about government services in general, and the board votes on one specific regressive sales tax. Those are related rather than identical, so the target is a proxy whose direction should hold and whose size is approximate. That does not explain two reversed signs.

It performs the persona instead of reasoning from it

The natural reading of a flat result is that the model ignores the persona. It does not. A classifier reads each trait back out of what members said, from held-out panels: age 0.948, income 0.931, never-rides-the-bus 0.990, stated attitude 0.923, sex 0.657, where 0.50 would mean no trace at all.

So age is plainly audible in the speech and moves the vote by −0.026 against a real gap of −0.189. Income is audible and moves it the wrong way. The model reads the demographic, voices it fluently, and then does not reason from it when it decides.

The coherence gate

Because none of that is visible in an outcome distribution, we built an audit: a larger model reads a smaller model’s turns and reports which side they argue, whether the reason is drawn from that person’s own circumstances, and whether the speech fits the background they were given. Held against the ballot they actually cast, it catches a member who argues one way and votes the other.

On 528 board members:

About a third of simulated members fail. One in nine argues one way and votes the other; one in four says something that contradicts the person they were told to be.

The two directions are not redundant. The judge reads the argument the same way with or without the background, but the rejection rates differ by 15.5 points, and that difference is entirely the persona-fidelity check. Roughly half the defects cannot be found by reading the transcript alone.

Repairing the discussion prompt so it names one question and says what is already settled took the on-topic rate from 53.6% to 99.9%, and took the agreement between a member’s words and their own vote from 0.605 to 0.885.

Where this leaves the pilot

Two findings we would defend, and one line of work that has outgrown this note.

Composition steers a simulated room through what members are told they believe, almost deterministically, and not through who they are told they are. A synthetic panel built from census demographics will not carry the real demographic gradient on a values question, and the error is directional rather than random, which means it can be checked in advance against any survey that covers the outcome.

The coherence gate is now its own project. It is a measurement instrument and a product feature at once, and its open questions are not answered here: whether a judge one size up beats a same-size judge, whether regenerating rejected turns changes the outcome or only tidies the transcript, and what an acceptable rejection rate looks like.