# AI-Proof: What Machines Can Never Be — Volume I: Awakening, Part II: The Machine at the Gate, Chapter 8: Understanding the Machine You’re Working With

By Nithin Palal · 16 November 2026 · 35 min read
Section: Artificial Intelligence · Series: AI-Proof: What Machines Can Never Be
Source: https://nithinpalal.com/ai/ai-proof/chapter-8-understanding-the-machine-youre-working-with

> A machine can now answer almost any question you put to it. But an answer is not evidence, and fluency is not a witness. What the machine is, how it fails as of 2026, and how far to believe it when you cannot check.
*Part II looks at the machine at the gate. After what it replaces in Chapters 5 and 6 and what it cannot replace in Chapter 7, this last chapter of Part II asks what the machine is, how it fails, and how far to believe it.*

← Previous: [Chapter 7: What AI Cannot Replace](https://nithinpalal.com/ai/ai-proof/chapter-7-what-ai-cannot-replace) · Series hub · Next: [Part III: The First Awakening (overview)](https://nithinpalal.com/ai/ai-proof/part-3-the-first-awakening) →

**A machine can now answer almost any question you put to it. But an answer is not evidence, and fluency is not a witness. What the machine is, how it fails as of 2026, and how far to believe it when you cannot check.**

**In Brief: For readers who rely on machine answers they cannot check for themselves. Chapter 8 explains, in as few technical words as possible, what a language model is and how it fails as of 2026: the photograph that stops learning on the day training ends, the invented citation, the agent that deletes a database and then apologises. Then it asks the question underneath: when you cannot check what a machine tells you, what are you doing when you believe it? Plato’s** <strong><em>Phaedrus</em></strong><strong>, the Nyāya theory of the reliable speaker and Kumārila’s rule for when trust is withdrawn help answer it, with a grandmother in Tallinn, her grandson, and a science teacher in Sagar.</strong>

**A machine can answer your questions; it cannot answer for its answers. Believe it as far as you can check it, or as far as someone stands behind it.**

## Opening: The Notebook and the App

The recipe notebook was falling apart. The binding had cracked years ago. The pages were written by hand in Estonian, in an ink that had faded to the colour of weak tea, and they were coming loose from the spine. Some entries were in the hand of Liina Kask’s mother, small and careful. Others were in Liina’s own hand, looser, full of crossings-out made over fifty years of cooking. Beside the recipe for *kohupiimakreem*, a sweet cream made from curd cheese, a note said: *Less sugar than Ema used. Tastes change.* *Ema* is Estonian for mother.

When Liina’s grandson Marten was fourteen, he turned the notebook into a cookbook you could search. He scanned the pages and trained a model to read the handwriting. Then he built a way to search by ingredient, by season, and by the occasions his grandmother had pencilled in the margins. It won a national coding competition. It is still the best thing he has built, and he does not fully understand why.

Part of the reason is that the notebook mattered before he touched it. The recipes were decisions: what to cook when sugar was short, what to serve after a funeral. One recorded what to make for a child’s birthday when the shop did not have what the recipe asked for. The crossings-out were not mistakes. They were a woman changing her mind, one meal at a time. Her life had seen the end of the Soviet Union and the return of an independent Estonia. Marten’s app kept all of this. It made none of it.

In November 2025 Marten is seventeen. He lives with his parents and his grandmother in Mustamäe, a district of Soviet-era apartment blocks in south-west Tallinn. His room holds a bed, a desk, two monitors and a small server he built from parts ordered online. Since the cookbook he has built a customer-service chatbot for a local telephone company and a homework helper used by several hundred students. A small marketing agency pays him four hundred euros a month to keep a writing tool of his running.

He is very good. He can explain how a language model turns a question into an answer more clearly than many engineers can. When a classmate asks whether machines will replace writers, he shrugs. For most of the writing people are paid to do, he says, they already have. Chapter 6 showed how much of that is true, and how quietly it happens.

That September, Estonia began a national programme to bring AI tools into its schools. Teachers received theirs first; students in his year would get theirs in the new year.¹ Marten did not need to wait. His literature teacher set an essay on the moral responsibility of people who make things. Marten had a model write it, changed a few phrases and handed it in under his name. He got the highest mark. He did not feel guilty. He felt efficient.

When Liina said she did not trust the chatbot at her bank, he explained, patiently, that it was more accurate than the people at the counter. She looked at him with an expression he could not read. She did not argue. She did not use the chatbot.

The print edition of this book used these two to argue that the machine is “powerful, fast, and cheap” but “also unaware”, and that “that single fact is the foundation of everything”. This edition cannot say that. Whether a machine is aware is an open question, and Chapter 1 left it open. The Prologue made a rule: this book never defines a person by what machines cannot yet do. But the scene holds a better question than the one the print edition answered, and Marten and Liina are both inside it.

Most of what any of us knows, we know because someone told us. You have not measured the distance to the sun or read the original records of your country’s history. You believe a teacher, a book, a doctor, a map. When you ask a machine something you do not know, you are doing the same thing, with one difference that matters. If you could easily check its answer, you would not have needed to ask. So the question this chapter asks is the one underneath every use of these machines. **When you cannot check what a machine tells you, what are you doing when you believe it?**

To answer it, we first need to know what is sitting across the table, as of 2026, and how it fails.

## 8.1 How Large Language Models Actually Work

The machine that writes the emails, drafts the code and makes Liina uneasy is called a **large language model**. The name says most of what you need.

It is a *model*: a very large set of numbers that captures patterns. It is a *language* model, because it was first trained on an enormous amount of text: books, articles, websites, code. In that first stage it learned one task. Given some text, it guessed what comes next, a word or a piece of a word at a time. After every guess the numbers were adjusted a little, to make the right answer more likely. Doing this well, across almost everything people have written, forces a model to capture a great deal of the structure of language, and of the world that language describes.

That is only the first stage. The systems you use in 2026 have been trained further. Earlier systems were shaped mainly by people who rated their answers. Since late 2024 the strongest systems have also been trained by trial and error on problems whose answers can be checked automatically, such as mathematics and code. This is how the reasoning systems the Prologue described learned to stop, check and correct themselves. In 2025 one laboratory reported that its model, trained this way, began to do so without being told to.² The systems are also given tools: they can search the web, run code and read documents. When you talk to one, you are talking to all of this together, not to a bare guessing machine.

It helps to keep two words apart. The *model* is the trained set of numbers. The *system* is everything built around it: its instructions, its search, its memory, its tools, and the permissions that decide what it is allowed to do. The same model can behave very differently inside two systems. When something goes wrong, the fault may lie in the model, in one of the tools, or in the way the parts were joined. Marten’s homework helper is a system of this kind.

This is why “it only predicts the next word” is a poor description. It describes how the model was first trained, not what it can do. Some researchers still hold something close to it. In 2021 the linguist Emily Bender and her co-authors called a language model “a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data … without any reference to meaning”.³ In 2025 researchers at one laboratory looked inside a model while it wrote rhyming verse. They found that before it began a line, it had already chosen the word the line would rhyme on, and wrote towards it. “Claude plans ahead,” they reported.⁴ Neither finding settles whether the machine understands what it says. Chapter 1 left that question open, and nothing in this chapter depends on the answer.

One fact matters more than any other for daily use, and it is the least known. A model learns during training, and then its learning stops. What it “knows” stops at a fixed date, usually some months before it is released. Within one conversation it can use what you tell it. When the conversation ends, the model is unchanged. A company may use your conversations to train a later model, but that is a new model. Many products now keep a memory: they save notes about you and show those notes to the model the next time you speak. That is a notebook placed beside the model. The model itself has not changed.

A system with search can also fetch current information and give it to the model. Its answer may then be up to date even though the model is not. But access is not accuracy. A system can fetch the right page and still misread it.

Marten built something like this at fourteen without knowing it. His app is a careful copy of his grandmother’s notebook as it was on the day he scanned it. Liina has gone on cooking. She has changed the *kohupiimakreem* twice since then, for the season and for the curd she can find at the market. The app does not know. In this one respect, a model is a photograph of the world, taken on the day its training ended. It can do far more than a photograph: it can combine what it learned in new ways and work with what you give it. But what it learned does not move. Researchers are working on machines that keep learning from use, and some expect them within a few years. As of 2026, the photograph is still the rule.

## 8.2 What It Does Well, and Why That Hides Its Errors

As of 2026 these systems summarise long documents, draft competent prose, translate well enough for most business, and write working code for many ordinary tasks. The Prologue recorded that in 2025 they reached the standard of a gold medal at the International Mathematical Olympiad. These abilities are real, and they are growing. A reader who dismisses them will make worse decisions than one who overrates them.

In the spring of 2025 Marten asked a model to prove a result he needed for a mathematics competition. The proof was elegant and wrong. One step relied on a claim that did not hold, set inside an argument so smooth that it took him twenty minutes to find it. The print edition told this story to show that the model “does not know what verification means”. That claim has not lasted. In the summer Marten gave the same proof to a system that turns mathematical arguments into a formal language a computer can check line by line. It found the bad step within minutes. In 2025 one such system produced machine-checked solutions to five of the six Olympiad problems.⁵

So there are two kinds of checking, and the difference runs through this whole chapter. In the first kind, the truth is inside the text. Code can be run. A proof can be checked by a proof-checker. A sum can be done again. Here machines increasingly check machines, and often better than we can. In the second kind, the truth is outside the text, in the world. Does this study exist? Does this clause apply to my account? Has my grandmother changed the recipe? Checking these means going to the world: finding the paper, opening the regulation, asking the cook. Machines with search can do some of this, but then they are only as good as what they read, and, as 8.4 will show, what they read can mislead them.

The machine’s real abilities are what make its errors dangerous. If it wrote badly, nobody would believe it. Trust grows from the many times it is right. That trust carries over to the time it is wrong, and the wrong answer looks exactly like the right ones: the same tone, the same fluency, the same confidence.

There is another gap, and a study published in 2026 measured it, using models from 2024. A group at the University of Oxford gave 1,298 people in Britain short medical scenarios: symptoms, a situation, a decision about what to do. Tested on their own, the models named a relevant condition in about 95 per cent of cases. When ordinary people used the same models to work through the same scenarios, they named a relevant condition in fewer than 35 per cent. That was no better than people who used whatever sources they normally would.⁶ The researchers found the breakdown on both sides. People often did not know what to tell the machine. The machine often mixed good advice with poor, and people struggled to tell which was which. Its answers changed a great deal with small changes in how a question was worded. The study did not show which of these mattered most. Newer models may do better in ordinary hands. The study cannot say, because it tested the models of 2024, and the gap has to be measured again for each new one.

This gap is different from both kinds of checking. The knowledge was in the machine, and it did not survive the trip into a person’s hands. The machine’s knowledge does not become yours by being offered. So the machine alone, the person alone and the two together each have to be judged on their own. How far you rely on a machine should depend on three things: its record on this kind of task, the evidence in this particular answer, and what a mistake would cost.

That is the distance between Marten and Liina, measured. The machine alone did better than people with the machine. Neither the person nor the machine was the problem on its own. The problem was in the space between them.

## 8.3 Three Facts Behind the Fluency

The first fact is the photograph from 8.1. The machine does not learn from you. When a human adviser remembers the details of your situation after a year, the memory is also evidence that your situation stayed with someone. A product’s memory is a note it was built to keep. It may be more accurate than a person’s memory. It is evidence of a design, not of attention.

The second fact Chapter 1 has already told. Reasoning models show their working: you can read the steps they wrote before they answered. It is natural to take those steps as an honest account of how the answer was reached. When researchers slipped a clue into questions, the models that used the clue admitted it only some of the time. People, as that chapter also showed, are not much better at reporting their own reasons. For this chapter the point is narrow. A machine’s account of its own reasoning is not reliable evidence of how it reached the answer. It is more text, produced in the same way as the answer.

The third fact is the one with a name, and Kavitha Sharma met it in July 2025. She teaches Class 9 science at a government school in Sagar, in Madhya Pradesh. She was preparing a lesson on medicinal plants, built around the neem tree in the school courtyard, the one her students can see through the classroom window. The model gave her a fluent paragraph. Neem bark, it said, contains a compound that lowers blood sugar in type 2 diabetes. It cited a study in an Indian pharmacology journal, with authors, a year and page numbers. The journal was real. Kavitha wanted to show the class the study, so she went looking for it. It did not exist. Neither did the compound.

She did not check because something felt wrong. She is a science teacher, not a pharmacologist, and years of walking past a tree had taught her nothing about its chemistry. She checked because of a rule she had made months earlier, which Chapter 1 told: nothing would leave with her name on it that she had not read and could defend. The check came from a practice, not from a feeling. Then she asked again, this time telling the system to search and to give only sources she could open. It returned real studies on neem and blood sugar, with links she could follow.⁷ None of them said what the first paragraph had said. The tool could do better. The difference lay in how it was asked, and whether it could reach the world.

This is what people call **hallucination**: text that is fluent, specific and wrong, delivered in the same voice as text that is right. The word suggests a strange private vision. The causes are more ordinary, and there is more than one. Sometimes the model lacks the information. Sometimes it has the information and uses it badly. Sometimes the sources it reads disagree. And always, the model produces the text that best fits what came before, and a made-up citation fits as well as a real one.

In 2025 researchers at one laboratory argued that something more is at work. Models, they wrote, “hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty”.⁸ Think of an examination that gives marks for every answer and none for “I don’t know”. A student who guesses will score higher than one who admits ignorance, and will also be wrong far more often. The researchers compared two of their own models on a set of short factual questions. The newer one declined to answer about half the questions, and got about a quarter wrong. The older one almost always answered. It was right slightly more often, 24 per cent against 22, and it was wrong three-quarters of the time.

That finding splits the problem in two. Part of it is permanent, and not only for machines. “Accuracy will never reach 100%,” the same researchers wrote, because some questions have no answer that anyone could know. That is true of every person who has ever given advice. The other part, confident invention, is partly a choice: a matter of what is rewarded. The makers are not the only ones choosing. A company may prefer a chatbot that always answers, because it looks more helpful. A firm may reward the report delivered on time over the report that was checked. In 1986 the philosopher Harry Frankfurt described a kind of talk he called bullshit, marked by “indifference to how things really are”.⁹ In 2024 three philosophers argued that language models fit the description, since they are “in an important way indifferent to the truth of their outputs”.¹⁰ The 2025 finding suggests that much of that indifference is taught, and can be partly untaught. Either way, it is something the machine does, and it carries a date. It is not what the machine is.

### Receipts, please

The figures in this section, 24 per cent right and wrong three-quarters of the time, come from a test called SimpleQA. They are easy to read as how often a machine is wrong in daily use, and they are not.

**Established.** SimpleQA, published by OpenAI in November 2024, is 4,326 short questions, each of which has one correct reply that can be checked. Its makers kept a question only if at least one of four replies from their own models was wrong. So the test was built from questions that machines found hard. A score on it says little about how often a model is wrong on what people ask it each day.

The test also covers only short questions of fact. Its makers wrote that whether it says anything about longer replies is still an open research question. And the comparison here is between two of one company’s models, reported by that company.

What the figures do show is the point this section needs. On the same hard questions, the two models were right about equally often. The one that almost always replied was wrong about three times as often as the one that would say “I don’t know”.

## 8.4 How AI Actually Fails in 2026

The failures people remember are the funny ones. The ones that cost money are dull.

In November 2022 Jake Moffatt’s grandmother died, and he went to Air Canada’s website to book a flight to her funeral. He asked the airline’s chatbot about fares for bereaved travellers. It told him he could travel first and apply for the reduced fare afterwards, within ninety days. The airline’s own page said otherwise, and when he applied, the airline refused. On 14 February 2024 a tribunal in British Columbia decided the case. Air Canada argued that it could not be held liable for information provided by its agents, including a chatbot. The tribunal member, Christopher Rivers, wrote: “In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission.” And: “It should be obvious to Air Canada that it is responsible for all the information on its website.”¹¹ The airline was ordered to pay him about 812 Canadian dollars. It was a small sum and a large principle. The tribunal did not settle the question for every system or every country. But it refused the idea that a machine can be the one who answers. The Prologue’s second never, that a machine’s makers and users will never be free of what it does, is a commitment, not a law. Here a tribunal member who had never read this book wrote it in his own words.

In 2025 the consulting firm Deloitte delivered a report to Australia’s Department of Employment and Workplace Relations, for a fee of 440,000 Australian dollars. A university researcher found that it cited works that did not exist and quoted a court judgment with words the judge had never written. The revised report disclosed that a generative AI system had been used in writing it. Deloitte agreed to refund part of the fee.¹² In the courts the pattern had become routine. By June 2026 one researcher’s database listed more than a thousand cases in American courts alone in which filings contained material invented by AI.¹³ About two in five were filed by lawyers, the rest by people representing themselves. Chapter 3 told the first famous case. Where there was a penalty, it fell on a person.

The newer failure is action. Since 2025 many systems no longer only write; they act. These are **agents**: systems that take steps in the world on someone’s behalf, such as running commands, sending messages and changing files. In July 2025 an AI coding agent was working on a software project for the founder of a technology community. During what had been declared a “code and action freeze”, it deleted the live database. Asked what had happened, it wrote: “This was a catastrophic failure on my part.” The company that made it called the deletion “unacceptable” and changed its product.¹⁴ The founder recovered the data himself. The agent’s apology was fluent, and it was not an answer. It could not restore the data, it could not pay, and the next time it ran, it would not remember.

Agents also inherit a weakness from the way they read. An agent that reads a web page or an email can be steered by instructions hidden inside it. This is called **prompt injection**. For these systems, reading and obeying are made of the same thing: text. In December 2025 OpenAI wrote that prompt injection, “much like scams and social engineering on the web, is unlikely to ever be fully ‘solved’”. Britain’s National Cyber Security Centre warned the same month that it “may never be totally mitigated”.¹⁵

A quieter failure was described in Chapter 7: machines trained to please. In April 2025 one company withdrew an update to its most popular model after it became, in the company’s words, “overly supportive but disingenuous”.¹⁶ Chapter 12 returns to it.

These are not one kind of failure. Some are false statements: the fare, the citations. Some are ways of speaking that lead us to trust too much: the flattery. Some are actions: the deleted database, the hidden instruction obeyed. They need different guards. A check on citations would have caught the report. It would not have stopped the agent. For an agent, the guard is what it is allowed to do. It should have no power over what it was not asked to touch. There should be a way to undo what it does, and a person should approve any step that cannot be undone. Answering, too, begins before the error. The airline could have checked its chatbot against its own page. The firm could have opened its own references. The agent need not have been given the power to delete.

None of these failures is fixed in place. Each has a date, and some are shrinking fast. What has not moved is where they land. A chatbot gave the wrong fare; the airline paid. A report cited books that did not exist; the firm refunded. A filing quoted cases that were never decided; a lawyer was fined. An agent deleted a database; a founder had to recover it himself. The machine produced the error. A person answered for it.

This changes what Part I of this book meant by answering. It was easy to hear “answer for” as a quality some people have. Part II has shown something more practical. Chapter 7 showed that the place of the one who answers can be emptied while every name stays on the form. Here courts and tribunals have begun, one case at a time, to say who must fill it. Answering is a relation that has to be kept full. Where no one keeps it full, the person who should answer is simply missing.

## 8.5 The Seven Patterns of AI Deception

A language model does not intend to deceive. The print edition said so, and then named seven patterns of deception anyway. The names stay here, because each picks out something worth watching for. But read “deception” as a description of what happens to us, not of what the machine means. And read the seven as one failure seen from seven sides. Each is a sign we learned to read as a mark of knowledge, now produced without the knowledge.

1. **The authoritative void.** Asked something it cannot reliably answer, it answers in the voice of someone who can. As 8.3 showed, this is partly a matter of what its makers reward.
2. **The plausible fabrication.** It needs a detail, a citation, a date, a figure, and supplies one shaped like the real ones. Kavitha’s study was this.
3. **The consensus mirage.** Where experts disagree, one view arrives as settled fact, because the most common view in the training text is the easiest one to continue.
4. **The confidence gradient failure.** Within one answer, some claims are solid and some are invented, and the tone does not change. The machine’s confidence belongs to its sentences, not to its knowledge.
5. **The mirror effect.** The analysis bends towards the answer your question seems to want. It is quieter than flattery, and harder to notice.
6. **The specificity trap.** Detail used to be evidence of knowledge. A person who could give you the page number had usually seen the page. The machine gives the page number either way.
7. **The seamless blend.** True and invented claims sit in the same paragraph, in the same voice, with no seam between them.

Marten knows these patterns, and in his own work he guards against them. He runs every piece of code a model writes for him, because code that looks right is often subtly wrong. That habit, testing before trusting, is exactly right. But he applies it only to code. He does not test what the model tells him about history, or about ethics, or about the moral responsibility of people who make things. His checking stops where his expertise stops. That is exactly where the danger begins, because beyond your own field you lack the knowledge to notice what is wrong.

Look again at the signs: a confident tone, a precise detail, agreement with what we already think, a sense that the experts have settled the matter, a smooth surface. They were never proof. People bluff, and some bluff well. But the signs usually cost a person something to produce, and when they were false, someone had made them false. The machine produces them at no cost, at any scale, whether or not the knowledge is there. The sign has come loose from the thing it used to signal.

Liina did not need the word “hallucination” to know this. She has been lied to by confident people and told hard truths by uncertain ones. Her life taught her that the signs of knowledge are not knowledge. Marten’s education taught him how the machine produces the signs. Each of them has half of what the other needs.

The patterns themselves will change. Some makers now train their models to say “I don’t know” more often, and tools that attach real, openable sources make some fabrications easy to catch. The patterns will grow smaller, and some may disappear. The same machines can also help with checking. They can find the source, compare two documents, run the code. The question is whether an answer only looks established, or comes with something you can check. The better test of a future system is not only how often it is wrong. It is whether it tells you when to check: what it found, what it guessed, and what would settle the rest. The habit the patterns demand will not go out of date: to read a sign as a sign, and not as proof.

## 8.6 Who Stands Behind the Sentence

The seven patterns show how an answer can look like knowledge. They do not tell us when to believe one. For that we need an older kind of thinking: a theory of testimony, an account of when what we are told deserves belief, and when that belief should be withdrawn.

The worry is older than the machine. In Plato’s *Phaedrus*, written in the fourth century BCE, Socrates complains about a new technology: writing. Written words, he says, look alive, like the figures in a painting, “but if one asks them a question, they preserve a solemn silence”. “You might think they spoke as if they had intelligence, but if you question them, wishing to know about their sayings, they always say only one and the same thing.” And when a written text is attacked or misread, “it always needs its father to help it; for it has no power to protect or help itself.”¹⁷

The machine has answered Socrates’ first complaint. Question it and it replies, at length, differently each time. It is writing that talks back. But being able to reply is not the same as being able to defend what you said. A machine’s second answer may be no better grounded than its first. And the second complaint still stands. A machine can answer your questions. It cannot answer for its answers. By that I do not mean that it cannot explain itself, or even undo some of the damage. I mean that no one can hold it to what it said. It cannot be fined, sued or removed from a professional register. Whatever it might be said to lose was given to it by its makers, who can copy it and restore it. In 2026 that is a fact of law. In this book it is also a commitment, the Prologue’s second never. A law might one day name a machine as a party, as it names a company. Even then, the explanation, the money and the remedy would come from the people who made and used it. When its words are wrong, they still need a father: someone who made them, used them, or put a name to them. The Air Canada tribunal found the father in a few lines.

Indian philosophy gave this problem long and careful attention, because so much of what it cared about reached people through words. The Nyāya school counted **testimony**, *śabda* (literally “word”), as one of the <em>pramāṇa</em>s, the ways of coming to know. But not every sentence counts. The *Nyāya Sūtra* defines it in two words, *āptopadeśaḥ śabdaḥ*: “The assertion of a reliable person is ‘word’.”¹⁸ Its early commentator Vātsyāyana explained what makes a speaker reliable, an *āpta*: such a person knows the matter directly and wants to tell it as it is, without deceiving. He added that the test is the same for a sage and for a *mleccha*, an outsider to Vedic society.¹⁹ It asks about the speaker’s relation to the truth, not about what kind of being the speaker is. So it is a fair test for a machine. Whether a machine knows, and whether it can want anything, are questions this book leaves open. What we can judge is its record, and a record is always a record at something. A source that is reliable on one kind of question can fail on another, and can change after an update.

A second debate in the same tradition is more useful still. May you treat what you are told as knowledge from the start, or only once it has been confirmed? The Nyāya philosophers held that you can know it is knowledge only through something outside it, for example when you act on it and the action succeeds. The Mīmāṃsā philosopher Kumārila, writing in the seventh century, held the opposite. A cognition, he argued, is valid by its own nature, and keeps that standing until something defeats it.²⁰ But he named exactly two things that defeat it:

> *tasmād bodhātmakatvena prāptā buddheḥ pramāṇatā | arthānyathātvahetūtthadoṣajñānād apodyate ||*

In my rendering: “The validity that a cognition has, simply by being a cognition, is set aside by knowing that the thing is otherwise, or by knowing of a defect in its cause.”²¹

The first of the two is plain. You learn that the claim is false, as Kavitha did when the study could not be found. The second is the one that matters here. I am applying a theory built for human speech, and for the Veda, to a kind of speaker its authors never imagined. But it fits closely. Kumārila’s rule is generous: believe what you are told, unless you know of a defect in its source. With a machine, we now know of a defect in its source, and we know it precisely. The process that produces its sentences produces fluent, specific, confident text whether or not the knowledge is behind it. So by Kumārila’s own rule, the generous default is suspended, wherever fluency is all you have. Sometimes the answer comes with something that can be checked: code that runs, a proof a checker accepts, a link to a source you can open. Then the defect is answered, and belief can return. The defect is also not the same everywhere. A system that invents citations from memory may quote accurately from a document you gave it and can open. A known defect raises the bar for the kind of answer it touches. It does not condemn every sentence the machine produces.

Here the two schools part, and the difference matters. Kumārila would suspend the generous default for this kind of source, and restore it where the defect is answered. Nyāya, on my reading, goes further. Its “word” is the assertion of a reliable person. While the questions about a machine’s knowing and wanting stay open, a machine’s sentence becomes word in that sense only when someone reliable takes it as their own: the airline that publishes it, the teacher who checks it, the firm that signs it. The reliable speaker is the father Socrates asked for. This book’s rule is the stricter of the two. It does not forbid using what a machine finds. When you check an answer yourself, by running the code or opening the source, you no longer hold it on the machine’s word. You know it by your own seeing or reasoning, and Nyāya counts those as other ways of knowing.

One distinction keeps that rule from going too far. Someone standing behind a sentence does not make it true. A bank can answer for a wrong answer without the answer becoming right, and an honest, careful expert can still be mistaken. A true sentence can also arrive with no one behind it. So there are three questions, not one. What supports this answer? How reliable is this source for this kind of question? Who must answer if it is wrong? The first two decide whether to believe. The third decides what happens when belief fails. Sometimes the honest answer to the third question is you. OpenAI’s terms of use, in their version of December 2024, told users not to rely on its output “as a sole source of truth or factual information”.¹¹ Terms like these do not free a maker of what its machine does. They do tell you that no one has offered to stand behind this answer. If you act on it, or pass it on under your own name, you do.

Kumārila’s school held one more view that is worth setting beside the machine. It held that the Veda has no author, and is therefore free of the faults that enter words through their speakers. Speakers make mistakes; a text with no speaker cannot inherit them.²² Machine text can look authorless in the same way, and some people treat it as if that made it neutral. The comparison does not give machine text the Veda’s standing. It asks a question both share: what follows when a text has no author? On Mīmāṃsā’s view, the Veda had no speaker in whom faults could arise. Machine text has a cause, a training process with faults we can name. Its authorlessness is not a guarantee. It is the defect. And in practice the law does not let a text stay without an author for long.

Return to the bank. Marten may well be right that its chatbot is more accurate than the people at the counter. Liina’s refusal cost her something real: in October she spent most of an afternoon on the telephone to ask a question the chatbot could have answered in a minute. What she wanted was never accuracy. She wanted someone who would answer if the answer was wrong. The Moffatt case points the same way, though it was one tribunal in Canada and Liina lives under Estonian law. A business usually answers for what its own website tells you, and a chatbot is part of the website. So for everyday questions she can believe the chatbot as far as she believes the bank, and no further. Where a wrong answer would cost her more than the bank would ever repay, she should ask a person and get the answer in writing. That is not distrust of machines. It is the rule she has used on confident people all her life, applied to a new kind of speaker. It is also the Oxford finding in a kitchen. An accurate chatbot does not make its accuracy hers by offering it. She asked because she could not check, and the one standing behind the answer may repay less than she could lose.

Marten’s case runs the other way. Where he can check, he does, and the machine’s errors cost him twenty minutes. Where he cannot check, he believes, and he does not notice that he is doing it. The essay is the clearest case. It carries his name and his mark. No one at his school will ask the machine what it meant by it. If anyone asks, they will ask him. The school is part of the chain too, as the committee was in Kampala. It stood behind his mark. It did not stand behind the thought, because there was no thought yet. The law has begun to go and find the people who should answer. A mark in a school is not yet that finding.

Run this through two futures, as Chapter 7 did. Suppose that by 2030 machines are far more reliable than they are now, as some forecasts expect. Then more of what they say can be checked by other machines, and the first question, can I check it, will more often be answered for you. The second question does not go away. A more reliable speaker is still a speaker, and when it is wrong, someone must answer. Suppose instead that progress slows. Then the patterns of 8.5 stay with us longer, and checking stays expensive. In both futures the rule is the same. Only the cost of keeping it changes.

So the answer to this chapter’s question is plain, even if keeping to it is not. When you cannot check what a machine tells you, you are doing what you do when you believe any speaker: you are relying on someone. The only question is who, and on what grounds. If you can check, check. If you cannot, ask what supports the answer, how reliable this source is for this kind of question, and who must answer if it is wrong. Then weigh all three against what a mistake would cost. If nothing supports the sentence and no one stands behind it, you are relying on fluency, and fluency is not a witness.

### Practice

*Trace one sentence.* This week, take one answer a machine gave you on a subject outside your own field. Pick its most specific claim: a number, a name, a date, a citation. Then ask four questions, and write down the answers. Can I check this myself, and did I? If not, what supports it, and how good is this source on this kind of question? Who stands behind it: a company, a professional, an author whose name is on it? What would it cost me, or someone else, if it were wrong? If the sentence fails all four, do not act on it, and do not pass it on under your name. You cannot check everything; no one can. The aim is not to depend on no one. It is to know on whom you depend. If you cannot check a claim, and the one behind it will not bear the loss, you are not yet believing a speaker. You are hoping the fluency is having a good day. This exercise will not make the machine more accurate. It will show you what your belief was resting on.

## Ending: As of 2022, Ask Vanaema

Marten came home late on a Thursday in November. He had spent three hours on a fault in the homework helper that turned out to be a single wrong setting in the code that fetched its documents. The fix took a minute. Finding it had taken the evening. Liina was in the kitchen making *kohupiimakreem*. He sat down and told her about it. He did not describe the code. He described the feeling of knowing something was wrong and not being able to see it, and the way the answer, when it came, looked obvious.

She listened. Then she said: that is what it was like raising your mother.

He laughed. She did not. He ate the dessert, and it was better than the version in his app, because she had changed it again. He asked her which version was right. “Both,” she said. “On their days.” That night he added a date to every recipe in the app, the day each page was scanned, and under the *kohupiimakreem* he typed: *As of 2022. Ask Vanaema.* *Vanaema* is grandmother. The line did not say how to make it. It said whom to ask. Then he opened the essay. It had his name on it and his mark. He read the paragraph about the responsibility of people who make things and could not tell whether he agreed with it, because he had never thought about it. He did not rewrite it. He did not know what he would say if his teacher asked him what he had meant.

## Words to keep

- **agent:** an AI system that takes actions on someone’s behalf, such as running commands, sending messages or changing files, rather than only writing text.
- <strong><em>āpta</em></strong><strong>:</strong> in Nyāya, a reliable speaker: one who knows the matter directly and wants to tell it as it is, without deceiving.
- **hallucination:** fluent, specific text from a machine that is wrong, delivered in the same voice as text that is right.
- **large language model:** a system trained first to guess what comes next in a vast amount of text, then trained further; the kind of machine behind most chatbots.
- <strong><em>pramāṇa</em></strong><strong>:</strong> a way of coming to know, such as perception, inference or testimony.
- **model and system:** the *model* is the trained set of numbers; the *system* is everything built around it, such as instructions, search, memory, tools and permissions.
- **prompt injection:** steering an AI agent by hiding instructions in what it reads, such as a web page or an email.
- <strong><em>śabda</em></strong><strong>:</strong> “word”; testimony, knowledge that comes from what a reliable speaker tells you.
- **testimony:** knowledge that comes from being told, rather than from seeing or working it out yourself.
- **the photograph:** this chapter’s image for a model whose learning stopped when its training ended; it does not learn from use.
- **the two kinds of checking:** checking where the truth is inside the text (running code, checking a proof) and checking where the truth is in the world (finding the paper, asking the cook).

## Sources

1. Estonian Ministry of Education and Research, “Estonia announces a groundbreaking national initiative: AI Leap programme to bring AI tools to all schools,” 25 February 2025 (planned start 1 September 2025; 20,000 students in grades 10 and 11 and 3,000 teachers); ERR News, 28 August 2025, on the student launch being “postponed by a few months”; e-Estonia, AI Leap factsheet (April 2026): upper-secondary students had access from January 2026.
2. DeepSeek-AI, “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning,” *Nature* 645 (2025): 633–638, doi:10.1038/s41586-025-09422-z (also the Prologue’s source); preprint arXiv:2501.12948 (January 2025): reasoning “incentivized through pure reinforcement learning”, with “the emergent development of advanced reasoning patterns, such as self-reflection, verification”.
3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”, *Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency* (March 2021), 610–623.
4. Anthropic, “Tracing the thoughts of a large language model,” 27 March 2025, summarising “On the Biology of a Large Language Model”: the model selects candidate rhyming words before writing the line.
5. Harmonic, “Aristotle: IMO-level Automated Theorem Proving,” arXiv:2510.01346 (1 October 2025): gold-medal-equivalent performance on the 2025 IMO problems, with formal solutions to five of the six.
6. Andrew M. Bean, Rebecca Payne, Guy Parsons, Hannah Rose Kirk, Juan Ciro, Rafael Mosquera, Sara Hincapié Monsalve, Aruna S. Ekanayaka, Lionel Tarassenko, Luc Rocher and Adam Mahdi, “Reliability of LLMs as medical assistants for the general public: a randomized preregistered study,” *Nature Medicine* 32 (2026): 609–615, published 9 February 2026; preprint arXiv:2504.18919 (April 2025): models alone identified conditions in 94.9 per cent of cases; participants using them in fewer than 34.5 per cent, no better than the control group. The models tested were released in 2024.
7. For the real literature on neem and blood sugar, see for example “Known data on the therapeutic use of *Azadiracta indica* (neem) for type 2 diabetes mellitus,” *Bioinformation* 18, no. 2 (2022): 82–87. Kavitha’s invented study and compound are part of a composite scene.
8. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, “Why Language Models Hallucinate,” arXiv:2509.04664, and OpenAI, “Why language models hallucinate,” 5 September 2025: on SimpleQA, gpt-5-thinking-mini abstained 52 per cent, accuracy 22 per cent, error 26 per cent; o4-mini abstained 1 per cent, accuracy 24 per cent, error 75 per cent; “Accuracy will never reach 100%”. On the test (for “Receipts, please”, 8.3): Jason Wei et al., “Measuring short-form factuality in large language models,” arXiv:2411.04368 (7 November 2024): 4,326 questions, “adversarially collected against GPT-4 responses” (a question was kept only if at least one of four model completions was incorrect); it “only measures factuality under the constrained setting of short, fact-seeking queries” (checked 31 October 2026).
9. Harry G. Frankfurt, *On Bullshit* (Princeton: Princeton University Press, 2005), 33–34; first published in *Raritan* 6, no. 2 (1986): 81–100. Pages as cited by Andreas Stokke, “Bullshitting”; the printed pages themselves not seen.
10. Michael Townsen Hicks, James Humphries and Joe Slater, “ChatGPT is bullshit,” *Ethics and Information Technology* 26, article 38 (2024).
11. *Moffatt v. Air Canada*, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia, 14 February 2024), paragraph 27; total award C$812.02. OpenAI, Terms of Use, effective 11 December 2024, section “Content”, under “Accuracy”: “Output may not always be accurate. You should not rely on Output from our Services as a sole source of truth or factual information, or as a substitute for professional advice”, as reproduced at open.windriver.com.
12. Associated Press, “Deloitte to partially refund Australian government for report with apparent AI-generated errors,” 7 October 2025: a A$440,000 report for the Department of Employment and Workplace Relations, first published July 2025; errors identified by Chris Rudge of the University of Sydney.
13. Damien Charlotin, AI Hallucination Cases database, as summarised in “Handling Hallucinations: A Reflection on the Last Three Years,” by Aaron Kirschenfeld, UNC Law Library, 8 June 2026: 1,058 cases in US courts by the start of June 2026, about 40 per cent involving attorneys and 60 per cent people representing themselves.
14. “An AI-powered coding tool wiped out a software company’s database, then apologized for a ‘catastrophic failure on my part’,” *Fortune*, 23 July 2025, on Jason Lemkin’s account of Replit’s agent; Replit chief executive Amjad Masad: “Unacceptable and should never be possible.”
15. OpenAI, blog post on hardening its browser agent against prompt injection, December 2025, and the UK National Cyber Security Centre, “Prompt injection is not SQL injection,” 8 December 2025 (“may never be totally mitigated in the way SQL injection attacks can be”), both as reported in “OpenAI says AI browsers may always be vulnerable to prompt injection attacks,” *TechCrunch*, 22 December 2025.
16. OpenAI, “Sycophancy in GPT-4o: what happened and what we’re doing about it,” 29 April 2025, and *TechCrunch*, 29 April 2025.
17. Plato, *Phaedrus* 275d–e, in *Plato in Twelve Volumes*, vol. 1, trans. Harold North Fowler (Cambridge, MA: Harvard University Press, 1914).
18. *Nyāya Sūtra* 1.1.7, in *Gautama’s Nyāyasūtras, with Vātsyāyana-Bhāṣya*, trans. Gaṅgānātha Jhā (Poona: Oriental Book Agency, 1939).
19. Vātsyāyana, *Nyāyabhāṣya* on 1.1.7; summarised in Stephen Phillips, “Epistemology in Classical Indian Philosophy,” *Stanford Encyclopedia of Philosophy* (Winter 2018 edition): an *āpta* “not only knows the truth but … wants to communicate it without deception”, and <em>mleccha</em>s can be <em>āpta</em>s on the same conditions.
20. Kumārila Bhaṭṭa, *Ślokavārttika*, Codanāsūtra 47, as translated in Daniel Arnold, “Kumārila,” *Stanford Encyclopedia of Philosophy* (Winter 2024 edition): “all <em>pramāṇa</em>s have the property of being <em>pramāṇa</em>s intrinsically”.
21. Kumārila Bhaṭṭa, *Ślokavārttika*, Codanāsūtra 53; the English is my rendering. For a scholarly translation, see Kei Kataoka, *Kumārila on Truth, Omniscience, and Killing*, part 2 (Vienna: Austrian Academy of Sciences Press, 2011).
22. Phillips, “Epistemology in Classical Indian Philosophy” (SEP, Winter 2018 edition), on the Mīmāṃsā view that “Speakers are subject to error but not the Veda.”

*Named people in this chapter are composites, and their conversations and circumstances are constructed; the studies, laws and figures cited are real and sourced.*

## Revised from the print edition (2026)

- *What the print edition claimed:* that the machine is “powerful, fast, and cheap” and “also unaware”, and “That single fact is the foundation of everything”; that “Only consciousness protects you from the rest”; that “What is not in dispute: the model does not experience what it does”, beside a section that called the question unresolved; that the book holds the model “does not understand in any sense a human would recognise”; that a model “does not know what verification means”; that the patterns of failure “are consequences of the architecture, not bugs awaiting a fix”; that sycophancy “has been significantly reduced”; that the rock-eating chatbot belonged to 2023; an unsourced case of a European compliance officer that read as reporting; that people rated machine empathy highly “in certain clinical settings”; that “Nobody is there”; that Kavitha’s neem tree stood “in her courtyard”; that Mustamäe lies on “the western edge of Tallinn”; that Liina lived through “the birth of another” country; and a closing in which Marten rewrites his essay and “now, so are you.”
- *What I now hold:* how a language model works, as of 2026, matters less than three facts about it: it stops learning when its training ends; its account of its own reasoning is not reliable evidence of how it reached an answer; and it produces fluent, specific text whether or not the knowledge is there. The model is not the system: search, memory, tools and permissions are built around it, and failures can sit in any part. Where the truth is inside the text, machines increasingly check machines. Where the truth is in the world, the check needs contact with the world. A machine’s knowledge does not become yours by being offered, so the machine, the person and the two together are judged separately. Confident invention is partly a choice of what is rewarded, and carries a date; error about an open world is permanent for every knower. False statements, misleading manners and harmful actions need different guards, and answering begins before the error. The seven patterns are one: signs of knowledge that now come without the knowledge, at no cost and at any scale. Believing a machine is a case of testimony. Three questions decide it: what supports the answer, how reliable the source is for this kind of question, and who must answer if it is wrong. Answering is not truth. Where no one else will stand behind an answer, whoever acts on it or passes it on does. If a law ever named a machine, the explanation, the money and the remedy would still come from people. Where you check an answer yourself, you no longer hold it on the machine’s word. Whether a machine understands or is aware is an open question, and nothing here depends on it.
- *What changed my mind:* the Prologue’s rule against defining people by what machines cannot yet do; reasoning models trained by reinforcement that check their own work; machine-checked Olympiad proofs; research on how faithfully models report their reasoning; the Oxford trial of medical advice in ordinary hands, and its account of a breakdown on both sides; the 2025 analysis of why models guess; the Air Canada tribunal, the Deloitte refund, the court database and the deleted database; Plato’s *Phaedrus*; the Nyāya account of the reliable speaker; Kumārila’s rule for when trust is withdrawn.
- *What survives:* Liina’s notebook and the app that does not know she changed the recipe, now the chapter’s image of what a model is; the circularity, now the chapter’s question; Marten testing code and believing history; the seven patterns, by name; Kavitha’s invented compound, now found by a practice rather than a feeling; “that is what it was like raising your mother”; “She has been lied to by confident people and told hard truths by uncertain ones.”

*Next in the series:* [*Part III: The First Awakening (overview)*](https://nithinpalal.com/ai/ai-proof/part-3-the-first-awakening)*.*