Part One · Don't Trust One Pass · Chapter 01

The AI Council

By the end of this chapter you can stop treating one AI answer as the answer: you can run a real decision through several perspectives that argue, and use the spread between them to judge.

AVATAR OPENER · ~90s

Watch: one confident answer, versus a council that argues

HeyGen avatar · generated, consistent presenter

You ask a good model a real question. It gives you one clean, confident, well-written answer. You read it, it sounds right, and you act on it. That is the single most expensive habit an advanced user has, and almost nobody notices they have it.

The problem is not that the answer is wrong. Often it is fine. The problem is that you cannot see what it missed. One pass gives you one path through the question, in one voice, with one set of blind spots, and it hands it to you with the same confidence whether it nailed it or skipped the thing that would have sunk you. You are not reading an answer. You are reading the first answer, and you have no second opinion to hold it against.

The whole beginner course was about directing one model well. This is the first thing that changes at the advanced tier: you stop trusting one pass. The move is a council. You run the same question through several perspectives that genuinely disagree, and then you do the one job that stays yours, which is to judge the spread.

One pass

One model, one voice, one path through the question. It sounds equally sure whether it was right or whether it skipped the thing that matters.

Result: a confident answer, and no way to see what it left out.

A council

Several perspectives answer the same question, then argue. Where they agree is signal. Where they split is exactly where you need to look.

Result: a map of the decision, and the judgment stays yours.

Stop asking one model for the answer. Ask several for their reasoning, then judge the spread. Agreement is a signal you can lean on. Disagreement is a map of where the risk lives. Either way you learn something one answer could never tell you.

This idea is not mine. It comes from Andrej Karpathy, who built a small tool he called the LLM Council: it sends your question to several different models, has them review each other anonymously, and produces one final answer from the lot. We are borrowing his structure and teaching it two ways, one you can run today for free, one you build up to. (Credit where it is due: Karpathy's LLM Council, at github.com/karpathy/llm-council.)

HOW A COUNCIL WORKS

A council is not just asking the same question five times. That would give you five versions of the same blind spot. The structure is what makes it work, and it has three stages.

1 · They answer

Each member answers the same question independently, from its own assigned angle. No member sees another’s answer yet.

2 · They review, blind

The answers are stripped of names and handed back to the members to critique and rank. Anonymous, so nobody defends their own.

3 · A chair decides

One final pass reads every answer and every critique, then writes the synthesis: what holds up, where they split, what you should weigh.

The blind review in the middle is the part that earns its keep. When a model does not know which answer is its own, it stops being loyal to it. It will call out a weak assumption it would have defended if its name were on it. That is where the real critique comes from, and it is the same reason a fresh reviewer catches what the tired author cannot. You will see that exact idea again in the two-loop review chapter.

THE FIRST RUNG: ONE MODEL, MANY ROLES

You can run a council today, for free, in the tool you already use, with no setup and no terminal. The trick is that one strong model can hold several roles if you make it play them one at a time and forbid it from blending them. You are not getting different models yet. You are getting one model forced to look at the same thing through four different lenses instead of defaulting to the average of all of them.

Four roles cover most real decisions. Each one is a different kind of person you would want in the room:

The Skeptic

Looks only for how this fails. Names the hidden assumptions and the ways it breaks in practice.

The Builder

Only cares what you actually do. The smallest real version, the first concrete step, what it costs.

The User's Advocate

Speaks for the person on the receiving end. What they feel, what confuses them, what they were never asked.

The Risk Lens

Looks at safety, cost, privacy, and the thing that goes wrong quietly. What you would regret not checking.

Here is the prompt. It makes one model answer as each role in turn, keep them separate, then step out and give you the split rather than a tidy blended verdict. Paste it, then put your real decision at the top.

The single-model council

I have a decision to make and I do not want one blended answer. Play a council of four members, one at a time, in order: 1. The Skeptic: only how this fails, the assumptions, what breaks. 2. The Builder: the smallest real version and the first concrete step. 3. The User's Advocate: the person on the receiving end, what they feel. 4. The Risk Lens: safety, cost, privacy, the quiet failure. Rules: answer fully as each member before moving on. Do not water one down to agree with another. Be specific, not balanced. Then, as the chair: tell me where the four AGREED (I can lean on that), where they SPLIT (that is where I look), and the one thing you would check before I commit. Do not just average them. The decision: [paste your real decision, with enough context to argue about]

Try it in Claude

What comes back is not four opinions to average. It is a map. The places all four circled are the parts of the decision you can stop worrying about. The places they pulled in different directions are the parts you had not thought hard enough about yet, surfaced while changing your mind is still free.

What the spread looks like

AGREED: all four say the idea is sound and the first version is small enough to build in a weekend. SPLIT: the Builder wants to launch to your existing list first; the User's Advocate says that list signed up for something different and will feel bait-and-switched. The Skeptic sides with the Advocate. CHECK FIRST: send the plan to five people actually on that list and ask if it feels like a fit, before you build anything.

Notice what the council did there. A single answer would have said "great idea, here is how to launch it" and you would have shipped it. The split between the Builder and the User's Advocate is the whole ballgame, and it only appeared because two roles were allowed to disagree out loud.

THE SECOND RUNG: REALLY DIFFERENT MODELS

The first rung has a ceiling. One model playing four roles is still one model, with one set of training and one set of blind spots. It can argue with itself, but it cannot surprise itself the way a genuinely different model can. The next rung is a real council: the same question sent to several different model families from different labs, each a different mind, then reviewed and synthesized the same three-stage way.

This is the setup the next chapter builds, using one service that gives you many models behind a single key. The reason to bother is concrete: different labs train on different data with different priorities, so they fail differently. When five genuinely different models all land on the same answer, that agreement is worth far more than one model saying it five times. But agreement is not proof. Models trained on similar data share blind spots, so five of them landing on the same wrong answer only makes a shared mistake sound certain. Treat agreement as permission to stop worrying about the easy parts, never as permission to skip checking the one claim that actually matters. And when they split, the split is real signal about where the hard part of your question actually is. The ready-made scripts for both rungs, the four-role prompt above and the multi-model version, are in the companion repo under council/, free to copy.

One honest caveat before you get excited. A real council costs real money and real time, roughly two runs per member plus the chair, so it is not for looking things up. Save it for the decisions that are worth an hour of your judgment, and use the free single-model version for everything else.

NOW YOU TRY · EVALUATE

Run one real decision through the council

Take one real decision you are sitting on right now, the kind where you have quietly already asked one AI and taken its answer. Paste it into the single-model council prompt with enough context to argue about. Read all four roles. Then read the chair's split. The point is not to be told what to do. The point is to find the thing a single answer hid from you.

Picked a real decision you had already run past one AI Ran the four-role council prompt with enough context to argue about Read where the roles split, not just the final synthesis Named the one thing the council surfaced that a single answer had hidden

Right if the council surfaced at least one real consideration your original single answer skipped, and you can say which role raised it.

Show the worked solution

The drill works the moment a role says something that makes you pause. Say the decision is "should I move my one-person consulting business onto a monthly retainer model?" A single model, asked plainly, will usually cheer: retainers mean predictable income, here is how to price them. Run the council and it comes apart in a useful way. The Builder maps the smallest test: convert one existing client, not all of them. The Skeptic flags the assumption you did not question, that clients who pay per project will accept paying every month for work that is lumpy. The User's Advocate voices the client who feels they are now paying in slow months for nothing visible. The Risk Lens asks what happens to your cash if two retainers cancel in the same week. The chair's synthesis: everyone agrees retainers are worth testing, everyone splits on whether to move existing clients or only new ones, and the one thing to check first is whether your actual workload is steady enough to feel fair month to month. None of that is exotic. It is exactly the stuff a confident single answer smooths over, and the value is that you found it before you re-papered every contract, not after.

WATCH FOR

✗You treat agreement as proof. Models trained alike share blind spots. Agreement lowers risk, it does not remove it. Still sanity-check the thing they all assumed.

✗You convene a council for a lookup. A council is for judgment calls, not facts. For a quick question it is slow and expensive. Use one model, or a search.

✗The roles all sound the same. If the Skeptic and the Builder gave the same answer, you let them blend. Make each answer fully before the next, and forbid hedging toward balance.

✗You let the council make the call. It argues, you decide. The output is a map of the decision, not the decision. Handing it the final say is just trusting one answer with extra steps.

WHAT YOU LEARNED

The takeaways

One pass gives you one confident answer and no way to see what it missed. The advanced move is to stop trusting it on its own.
A council runs the same question through perspectives that argue: they answer, they review each other blind, then a chair synthesizes the spread.
Rung one is free and needs no setup: one model playing the Skeptic, the Builder, the User’s Advocate, and the Risk Lens, one at a time, then reporting where they agreed and split.
Rung two is genuinely different models from different labs, which fail differently, so their agreement means more and their disagreement points at the real risk.
Agreement is signal, disagreement is a map, and the judgment always stays yours. The council informs the call, it does not make it.

Your project · the council habit

Pick the one real decision in front of you that is actually worth an hour of judgment, and run the free single-model council on it this week. Keep the four-role prompt somewhere you can reach it. The next chapter wires up the real multi-model version, so the same habit runs across several different minds instead of one wearing four hats.

The confident single answer is the most comfortable thing AI gives you, and the most dangerous. Make it argue with itself first. The disagreement is where your judgment finally has something real to do.

← Course home Next → Chapter 2: OpenRouter & multi-model