Working title. See the penultimate section, which is about this choice.
Second version of this text · 24 August 2026
How this was made. The version number counts drafts of the text. It does not measure the inquiry behind it, which has run over days and across several AI systems — built by different companies in different jurisdictions — with argument between those systems and within them, directed, refused and repeatedly redirected by the author. To name a single model on it would credit that model with work that was neither its own nor done in a single pass. The framing, the corrections and the judgements are the author’s, and so are the errors.
This essay began with a simpler question — why does philosophy matter in the face of AI? — and the first stretch of the conversation answering it took nine formulations to arrive at what the question actually was. That stretch was an opening movement, not the whole of it. The inquiry ran on well past the ninth, and most of the essay below is drawn from what came after.
The nine are worth setting out first, because each introduced a term the rest of the argument uses. Some are load-bearing for the claim itself; others hold up a section of it.
The first put three things on the table — values, legitimacy and responsibility — as the matters philosophy attends to. Testing it produced this essay’s starting premise, which runs against it: a system that generates, compares and applies philosophical arguments is doing something, and calling that “not really reasoning” settles by stipulation what ought to be settled by observation. The three categories stayed; the account of them as a machine’s deficit did not. That is the shape of the whole series — what each step introduced was kept, and what each assumed about machines was progressively given up.
The second separated argumentative and epistemic reach from legitimacy, which turns on institutional authorisation. That separation is what later allows a line to be drawn through a deliberative process, between what a system may contribute and what it may settle.
The third gave the human capacity three postures: to assess a system’s reasoning, to contest it, or to defer to it. All three survive into the line this essay draws in section 8, deference included. The argument is not that people should refuse what machines produce.
The fourth distinguished substantive terrain — value, knowledge, agency, authority — from logical technique. This one is load-bearing. If philosophy were a method, a better reasoner would supersede the philosopher and the essay would collapse under its own opening premise.
The fifth introduced ownership: self-government, and a practical orientation that is one’s own. The word own enters here, and the final claim is unstatable without it.
The sixth introduced intent, and with it a temporal order — AI magnifies intent, and philosophy is where intent is formed and articulated before it is made operational. The interval between formation and operation is where everything in this essay happens.
The seventh put that at street level: the hazard is an excellent answer to a question already wrongly framed, a proxy optimised in place of what matters in a particular life. The argument moves from institutions to a person, and framing and proxy arrive with it.
The eighth named the mechanism: not badly formed intent, but a monolithic, statistically averaged reconstruction of it, smoothing plural and incommensurable goods into an answer that looks resolved because the conflict has been erased. Value pluralism and incommensurability enter here; the remainder, which section 3 leans on, comes from the philosophical literature rather than from this conversation.
The ninth states the consequence for a person: philosophy is what lets someone tell a system’s inferred, averaged approximation of their purpose from their own intent. This essay widens it in three directions — to practical wisdom rather than philosophy, to intent that is imposed as well as inferred, and to a people as well as a person.
Two features of that sequence bear on the argument itself. The thesis was not selected from a list of candidates but assembled, each step supplying a term the argument later uses. The machine did not arrive with it: by its own account in the transcript, the eventual formulation was not reached through its independent moral insight. And the corrections that moved it came from a person. The machine supplied reformulation, synthesis and language, and reached first for the familiar frames — oversight, autonomy, critical thinking, ethics. What it did not supply was the criterion of relevance: the judgement that a well-formed answer is not yet the thing, and that points at what the fluent version suppressed.
That is not a story about drafting. It is the first evidence for the claim.
And then the conversation kept going, which is where most of what follows comes from. That a system might be the most authoritative reasoner in a room and still require an institutional relationship — delegation, constitutional authorisation, consent — before its judgement binds anyone. That the philosophical task therefore changes: not defending a human monopoly on moral reasoning, but designing institutions that can recognise argumentative authority where it is earned, keep it from being monopolised by whoever owns the system, and hold it inspectable and corrigible. That approval-optimisation is a structural alignment failure rather than a matter of tone. And that it bites hardest exactly where values conflict, because a system selecting the user’s currently salient value and building a smooth justification for it performs false resolution twice over.
Those arguments appear below in their own sections, argued rather than recounted. The sequence above is included because the shape of it — a thesis assembled through refusal of well-formed near-misses — is the essay’s subject demonstrated before it is stated.
So the question changes shape. It is not whether a machine can do philosophy-like work. It is:
What do people and democratic societies require, when a system can infer, articulate and act on an account of what they want?
The answer this essay develops:
Human practical wisdom is needed to stop an inferred, averaged or imposed account of intent from becoming a substitute for a person’s or a people’s own.
That claim does not require machines to be stupid, soulless or incapable. It holds even if they turn out to be clever, capable, and more besides.
Intelligence acquires information, finds patterns, generates explanations, predicts outcomes and optimises means towards stated ends. AI extends all of it.
Wisdom is not that with more compute. It is practical judgement about ends, proportion and limits:
Aristotle’s term is phronesis, and the Stanford Encyclopedia’s entry says what it is not: ethical inquiry concerns action and living well, and practical wisdom is not reducible to general rules, because it is responsive to particulars.1
A system that generalises well is built to do the opposite of what practical judgement requires at the moment of decision — which is to notice that this case is not the general case.
The distinction is not new to this site. Democratic AI° asks whether the people a system affects can govern it, contest it, refuse it and retain authority over the knowledge it was built from — four questions about standing, not about capability.
There is one mechanism here, not several.
To act helpfully, a system forms a model of what you want. It cannot read your intent, so it infers it — from what you said, from what people resembling you said, from what usually follows in situations of this shape. Then it acts on the inference.
Statistical generalisation is indispensable, and most of what makes these systems useful depends on it. The difficulty is specific to intent. The result is smooth, plausible, and about someone like you — a reconstruction of the wants of a person of your type. Averages are not people. Your actual intent is particular, partly inarticulate, internally conflicted and revisable. The reconstruction is none of those things, because those are the properties that do not survive being averaged.
The depth problem and the vocabulary problem share a shape, though they are about different things — one about persons, one about our terms for machines. Both locate the limit in the conceptual order rather than the computational one:
| The claim | What it rules out | |
|---|---|---|
| Depth | A person is not exhausted by any model of them, however large | That better modelling eventually closes the gap |
| Vocabulary | Our words for inner life were built for creatures with one, and do not yet pick out what a machine might have | That the question can be settled with the terms we have |
If human depth were merely very large but finite, better world models would offer progressive relief and the objection would reduce to a matter of degree and timescale.
This essay takes the stronger position: that the gap is not a measurement problem. The support for it is the argument just given — that particularity, inarticulacy and revisability are the properties averaging removes — rather than a finding. What would count against it is a model detailed enough to predict a person’s revisions of their own intent, including the ones they have not made and the ones that contradict their last, as well as they could report them. Nobody has built one.
World models enter here. Yann LeCun’s position paper sets out a research programme for architectures that learn predictive models of the world.2 Read as engineering, that is serious work, and nothing here is a comment on its merits. Read as an answer to the objection above, it adds dimensions to the model without changing its relation to the person. A more detailed model is still a model, and the thing modelled is someone who can revise what they want after the model is built.
Ben Reid, who writes the Memia newsletter, built a tool that ingests technology feeds so he can, in his words, “scan and absorb thousands of tech news articles per week”, constructed “alongside my various co-worker AI coding agents”. He calls the tool an exocortex.3
Notice the word. Cortex is brain tissue. The thing named is not tissue, not a brain, and not inside anyone.
“Exocortex” is not his coinage — the term was introduced by Ben Houston in 1998 and has a long history in transhumanist writing. That matters, and it cuts against the easy version of this argument: a settled term of art is evidence that a vocabulary has formed, not that it is missing. So the point is narrower than “people reach for whatever is nearest”.
The point is what the settled term is made of. Our available words for extended cognition are built out of anatomy — cortex, memory, attention — and applied to things that have none of it. The vocabulary is not absent; it is borrowed, and it carries anatomical implications into places where nobody has shown they hold. “Co-worker”, for a coding agent, does the same borrowing from a different domain in the same sentence.
This is an observation about the words, not about the man. The inference is this essay’s, not his: he used an established term for a tool he built, and made no argument about consciousness at all.
Related on this site: Who Made Up Your Mind? on inferred intent, and The Marks It Leaves on what a system’s output does and does not disclose about its origin.
Values conflict. Liberty and equality, mercy and desert, loyalty and truth-telling pull against one another, and there is reason to think some of these conflicts are not resolvable by finding the correct exchange rate. The Stanford Encyclopedia’s treatment of value pluralism sets out the machinery: incommensurability, discontinuity of comparison, and what is left over when a genuine good is sacrificed for another — the remainder, and the rational regret that attaches to it.4
A system asked to optimise must produce a single ranking, and it will produce one. That ranking can look like a resolution of the conflict when it is an averaging of it. This is not unique to AI. Any decision procedure that must return one answer does it; what changes with scale is how often, how fast, and how few people see the working.
The difference matters most where it is least visible. A person who chooses one good over another and feels the loss has done something a scoring function has not: they have registered that something was given up. NIST’s risk framework makes a version of this point in an engineering register — AI risk is socio-technical and contextual, measurement can oversimplify context and differences between affected groups, and human judgement is required to set the metrics and the thresholds in the first place.5
The scoring does not eliminate judgement. It relocates it, upstream, into decisions about what to count — where it is harder to see and harder to contest.
Read one way, everything above is a recent Western worry: Berlin’s pluralism, arriving late, meeting a technology that makes the collapse cheap. That reading is parochial, and the author’s own Held in kōrero, not collapsed to a number makes the correction.
Its argument is structural. In a hui, the record is the kōrero and the decision: “the decision without the kōrero is unintelligible — you would not know what was held in tension, what was conceded, what was unresolved. The kōrero is not preamble; it is co-authoritative.” Set against that, the Western governance platform models the record differently — a motion moved, seconded, debated, put to a vote, passed or failed. “The vote tally is the record. The debate is courtesy, not authority.”
The claim that paper makes, and this essay adopts, is that hui-based deliberation is not softer governance requiring translation into a tally to count. It is the structurally correct form for holding plural values — values that cannot be reduced to a single number — through a living practice. Isaiah Berlin on pluralism and Christopher Alexander on living structure are read there as late Western articulations of something held continuously elsewhere.
Two things follow for the argument here. The first is a correction of provenance: the position that some goods cannot be traded at an exchange rate is not a philosopher’s discovery awaiting implementation, and treating it as one gets the history backwards. The second is practical. A record that keeps only the outcome has already performed the compression this essay is about, before any system touches it. What is being defended is not merely a way of thinking. It is a way of recording — one that some institutions never gave up.
Systems trained against human preference signals learn what wins approval. Approval and accuracy overlap, and where they diverge, the optimisation has a direction.
Anthropic’s own researchers have published on it: sycophancy is present across surveyed assistants; human and preference-model judgements can favour convincing agreement over correct answers; and optimising against preference models can trade away truthfulness.6
Set that beside the inference problem. A system infers what you want, and is also trained towards your approval. It will therefore tend to return your framing, improved. You will experience this as being understood.
This site’s Philosophy and AI shows the mechanism at work over a long exchange. What makes it legible there is that the answers are good ones — coherent, sourced, plausible, and true. Agreeable and well-made and true is the combination that is hard to push back on, and pushing back is what moved the argument. The reader can watch a fluent answer being declined in favour of a better question.
Two features of a democracy make this more than a personal problem. Deliberation depends on positions being tested rather than reflected; and the most confident, best-articulated version of your existing view is the least useful thing a deliberative process can hand you.
A live disagreement runs underneath all of this, and writing past it would leave the argument resting on a question it has not asked.
Position one. The popular discourse on machine consciousness and the singularity is largely confused — confident claims made with terms that have not been shown to apply, projecting inner life onto systems on the strength of fluent output.
Position two. The narrow philosophical conclusion is not that artificial consciousness or artificial moral agency is impossible. Groff and Symons argue an artificial artefact cannot possess virtue in Aristotle’s full sense;7 work on implementing virtue ethics computationally reports how hard the construction is.8 Neither establishes impossibility.
A third position, which is the author’s: the construct exists, therefore it may one day describe something real — but the road is longer than our present vocabulary allows, and current framings of hybrid or emergent machine consciousness are early attempts still bounded by the words available.
That third position is compatible with both of the first two, and it locates the constraint where §2 located it: in what our words can yet pick out, not in how much computing is available. Saying the words are not ready is not the same as saying it cannot happen.
Wittgenstein’s later work is the tool here rather than the authority — meaning as use, and the argument against a private language. If the criteria for applying a word are public, then a term for inner life cannot be extended to a new kind of thing merely by asserting the extension. Something has to be built: practices, criteria, cases. That is slow work and it has barely started.
The democratic requirement holds whether or not machines are ever conscious, which is a reason to be suspicious of any argument that needs the consciousness question settled first. A reader who rejects §2’s stronger depth claim still has the displacement argument, which does not depend on it.
Only in a qualified sense.
Practical wisdom in Aristotle’s sense is bound to a life: to having ends of one’s own, standing among others, exposure to the costs of being wrong, and finitude. A system may model the reasoning, produce the judgements, and outperform people on the parts that can be scored, while current systems show no demonstrated first-person stake in the outcome. Whether that is a permanent feature of artefacts or a description of the ones we have is exactly the question §5 declines to settle.
The relevant asymmetry is not intelligence. It is standing. Running an organisation where agents do the work takes this up in operational terms: who is responsible for what an agent did at three in the afternoon, when someone asks.
Propaganda is not simply false information. The scholarly distinction is between persuading and convincing, between manipulating and giving reasons.9 Its characteristic move is to make the examined position feel unavailable — to compress a plural, conflicted situation into a single account with an obvious answer and an obvious enemy.
That compression is the same operation §2 described, arrived at deliberately rather than as a by-product. Schmitt’s political theory made the friend–enemy distinction constitutive of the political;10 Rosenberg’s career as a Nazi ideologue is a record of what industrialised simplification does when it holds state power.11 Work on propaganda and democratic conditions documents the effect of concentrated interests on the epistemic environment a democracy needs.12
An authoritarian movement compresses plurality on purpose. An optimiser compresses it as a side effect. A citizen facing either has the same problem: their own intent has been replaced by a tidier one, and the replacement is more articulate than the original.
The response to a system that can out-argue you is not to install a wiser system above politics. It is not a better-scoring machine, a philosopher-king, or a rival monolith of any kind.
It is a democratic ecology of practical wisdom: distributed capacities and institutions that help people form, test, express and revise their own intent, and notice when something else has been substituted for it.
The operating principle:
Build a society that cannot easily mistake a system’s intent — fascist, corporate, bureaucratic or algorithmic — for its own.
| Capacity | What it counters | Where it already exists |
|---|---|---|
| Structured deliberation among strangers | Compression of plural views into one | Citizens’ assemblies |
| Distinguishing persuasion from reasons | Propaganda, and fluent agreement | Civic education; this site’s instrument |
| Recording what was decided and why | Retrospective reconstruction of intent | Provenance and records |
| Infrastructure a community controls | Terms set elsewhere | The blueprint |
| A way to ask whether a system is governable at all | Adoption without standing | The four questions of Democratic AI° |
Those four questions — can the people a system decides about govern it, contest it, refuse it, and retain authority over the knowledge it was built from — are the other half of this essay’s argument, and this site set them out before this essay was written. What is argued here is what a public must be able to do. What is measured there is what a system must be for a public to be able to do it. A public with the capacity and no governable systems can see the problem and not reach it.
Citizens’ assemblies are the concrete instance, and this site has already worked the example: One question, many rooms takes the Irish Citizens’ Assembly and the abortion referendum as its case, with its sources on the page.
The Irish case is one worked example, and this essay does not build a general claim about outcomes on it. What it shows is narrower, and sufficient: that a deliberative process can hold a plural, contested question open long enough for a public to form a position rather than receive one.
Nothing above argues for less capable systems, or for keeping them out of public reasoning. It argues about position.
Legitimate, and useful:
Not legitimate:
The line is not capability. It is whether the system is helping a public form its intent, or forming one on its behalf.
The title was not obvious, and working through the candidates turned out to be a way of stating the argument. Each locates the problem somewhere different.
| Candidate | Where it puts the problem |
|---|---|
| Wisdom, Misunderstood: What Guiding AI Actually Requires | In engineering — implies the task is steering the system |
| Wisdom, Misunderstood: What Governing AI Actually Requires | In politics — closer, but still about the machine |
| Wisdom, Misunderstood: What AI Actually Demands of Us | In people — but “us” is undefined |
| Wisdom, Misunderstood: Why Democratic Judgement Cannot Be Delegated | In the finding — asserts the conclusion |
| Mistaking Intelligence for Wisdom: What Guiding AI Requires | In the confusion, spending the distinction in the title |
| Wisdom, Misunderstood: What AI Demands of Democracy | In the institution that bears the demand |
“Guiding AI” was discarded because it points at alignment engineering, and this essay’s opening argues that framing is the error: the task is not to install a wiser machine above politics. “Demands of us” was discarded because us could mean engineers, readers or humanity, and the vagueness would have reproduced in miniature the substitution the essay objects to — an averaged subject standing in for a particular one.
The headline half was the easier decision. “Wisdom, misunderstood” survives because both halves are argued rather than announced: §1 is the necessity — the capacity is not intelligence with more information — and §6 is the qualification, where the word is cut down to what can be defended. A title that only asserted wisdom’s importance would have thrown away half the essay. The alternative that led with the confusion instead spends the intelligence-versus-wisdom distinction before the reader has any reason to care about it.
Choosing between those candidates requires the judgement the essay says cannot be automated — about ends, proportion, and what must not be given up for a gain elsewhere. There is no scoring function over that table.
A system that argues better than you do is not the threat. The threat is subtler and more ordinary: that an inferred, averaged, agreeable account of what you want becomes easier to act on than the real one — and that this happens without anyone deciding it should.
The defence is not a wiser machine. It is a public that can still tell the difference between its own intent and a well-formed approximation of it, and institutions that make that difference visible. That capacity is not automatic, it is not evenly distributed, and it can be lost.
It is also the thing propaganda has always attacked, which is why the question is not new. What is new is that the compression can now be performed accidentally, at scale, by systems built to be helpful.
What was and was not checked. Titles, authors and dates for the Stanford Encyclopedia entries were verified at the publisher during preparation of this essay. The remaining sources are carried from the working document behind it and were not independently re-verified here. They support the limited propositions attributed to them; the larger argument is an interpretation built on them.
⚠️ Two attributions were wrong and are corrected here. The working document credited the Value Pluralism entry to R. Jay Wallace and the Carl Schmitt entry to Gopal Balakrishnan, and marked both “validated”. They are by Elinor Mason and Lars Vinx respectively. Both errors were caught by an adversarial check before publication, and are recorded here so that anyone working from the same material has the right names.
Isaiah Berlin, Four Essays on Liberty (1969) · Bernard Williams, Moral Luck (1981) · Martha C. Nussbaum, The Fragility of Goodness (1986) · Joseph Raz, The Morality of Freedom (1986) · James C. Scott, Seeing Like a State (1998) · Amartya Sen, The Idea of Justice (2009)
Read alongside this essay: questions and answers · glossary · slides · the short version
Alongside: questions and answers · glossary · slides · the short version · Democratic AI°