NP Nithin Palal

Chapter 8: Understanding the Machine You’re Working With

Part II: The Machine at the Gate

· 21 min read · Discuss · Plain text

AI-Proof: What Machines Will Never Be — Volume I: Awakening, Part II: The Machine at the Gate, Chapter 8: Understanding the Machine You’re Working With

The recipe notebook was falling apart. The binding had cracked years ago, and the pages — handwritten in Estonian, in an ink that was slowly fading to the colour of weak tea — had begun to separate from the spine. Some of the entries were in Liina Kask’s mother’s handwriting, tight and precise, the penmanship of a woman who had learned to write under Soviet schooling and had kept writing in Estonian as a quiet act of defusal. Others were in Liina’s own hand, looser, more confident, annotated with crossings-out and marginal corrections accumulated over decades of cooking. A note beside the kohupiimakreem recipe said simply: less sugar than Ema used. Tastes change.

Liina’s grandson, Marten, turned the notebook into a searchable digital cookbook when he was fourteen. He scanned the pages, trained a model to recognise the handwriting, and built an interface that let you search by ingredient, by season, by the occasions his grandmother had pencilled into the margins. It was beautiful work. It won a national coding competition. It is still the best thing he has ever built, and he does not fully understand why.

The answer has something to do with the fact that the notebook mattered before Marten touched it. The recipes were not data. They were decisions — what to cook when sugar was rationed, what to serve at a funeral, what to make for a child’s first birthday when you could not afford the ingredients the recipe called for and had to improvise with what was available. The crossings-out were not errors. They were a woman adjusting her understanding of the world, one meal at a time, across a life that had included the collapse of one country and the birth of another. Marten’s tool preserved all of this. It did not create any of it. The difference between those two acts — preserving and creating, processing and understanding, the tool and the consciousness that directs the tool — is the subject of this chapter.

Marten is seventeen now and lives in Mustamäe, a residential district on the western edge of Tallinn. His bedroom contains a single bed, a desk, two monitors, and a server rack he assembled from components ordered online. Since the cookbook, he has built a customer service chatbot for a local telecom company, a homework assistant used by several hundred students across Estonia, and a content generation tool that a small marketing agency pays him four hundred euros a month to maintain.

He is brilliant. This is not flattery. He understands transformer architectures and the mathematics of token prediction with a fluency that many professional engineers would envy. He reads research papers for recreation. He can explain, clearly and accurately, how a large language model processes a prompt, assigns probability weights across possible next tokens, and generates text that appears — to most readers — indistinguishable from human writing.

When a classmate asks whether AI will replace writers, Marten shrugs: it already has, most people just haven’t noticed.

He is not entirely wrong. That is the uncomfortable part. The vast majority of professional writing — emails, reports, briefs, summaries, product descriptions, internal communications — is functional prose. It exists to convey information, not to move the reader. For this kind of writing, AI is not approaching human quality. It has matched it, and in many cases surpassed it in speed, consistency, and competence. The writers who produced this work are being replaced, not in some speculative future but now, in the same way that the clerks who once copied legal documents by hand were replaced by photocopiers. Marten sees this clearly, and what he sees is true.

What he does not see is the boundary. He does not see where the replacement stops and the irreplaceable begins. He does not see it because seeing it requires a kind of knowledge his technical education has not provided — not knowledge about the machine, but knowledge about himself. About what it means to struggle with an idea until the idea changes you. About the difference between generating a paragraph on the moral responsibility of creators and actually thinking about moral responsibility until the thinking leaves a mark.

When his literature teacher assigns an essay on exactly that subject, Marten uses his own AI tool to generate a draft, edits it lightly for style, and submits it. He receives an A. He does not feel guilty. He feels efficient. When Liina tells him she does not trust the chatbot at her bank, he explains, patiently, that it is statistically more accurate than the human tellers. She looks at him with an expression he cannot read. She does not argue. She does not use the chatbot.

This chapter is for both of them. For the Martens — who understand the mechanism and have not yet considered what it means. And for the Liinas — who sense something is missing but cannot name it in the language the technology demands. The first group needs to look beyond the mechanism. The second needs enough technical understanding to know what they are sensing. Both need each other, because understanding the machine you are working with — truly understanding it, not merely operating it or merely fearing it — is one of the most important acts of consciousness available to you right now.

8.1 How Large Language Models Actually Work

The technology that writes the emails, generates the code, passes the bar exam, and makes Liina uneasy is called a large language model. The name is more descriptive than it sounds.

It is large: frontier models contain hundreds of billions to trillions of parameters — numerical values adjusted through training to capture patterns in language. It is a language model: its foundational training process involves predicting what comes next in a sequence of text. And it is a model: a mathematical representation of patterns. Not a mind. Not a thinker. Not a knower.

During training, the model processes enormous quantities of text — books, articles, code, legal filings, scientific papers. It adjusts its parameters so that, given any sequence, it becomes extraordinarily good at predicting what should follow. This core process is called next-token prediction. A token is a piece of text — sometimes a whole word, sometimes a fragment. The model generates its response one token at a time, each chosen based on the probability distribution created by everything that came before.

But next-token prediction is only the foundation. By 2026, the models you interact with have been shaped by processes far beyond their initial training. They are refined through reinforcement learning from human feedback, which teaches them to follow instructions and be helpful. They are trained on synthetic data generated by other models. They are given tools — the ability to search the web, run code, retrieve documents. They reason through problems in extended chains of thought. The system you talk to is not a raw prediction engine. It is a prediction engine that has been extensively coached, constrained, and augmented. The gap between what the base model does and what the deployed system does is large, growing, and poorly understood even by experts.

This matters because the common simplification — that the model ‘just predicts the next word’ — misleads in both directions. It makes the technology sound stupider than it is, and it makes it sound more transparent than it is. The base capability is pattern prediction. The refined behaviour — the helpfulness, the apparent reasoning, the capacity to follow complex instructions — emerges from layers built on top of that base. Whether what emerges constitutes anything like understanding is one of the genuinely unresolved questions of our time. Serious researchers disagree. Some have argued that sufficiently advanced prediction may require, and therefore produce, something functionally equivalent to comprehension. This book takes a position — that the model does not understand in any sense a human would recognise — but it takes that position honestly, knowing the counterargument is not foolish.

What is not in dispute: the model does not experience what it does. It does not form an intention to help you. It does not feel the momentary confusion that precedes clarity. It generates. The result often looks like understanding because language, it turns out, contains far more structure than we assumed, and a system that masters the prediction of structure can produce outputs indistinguishable from genuine comprehension.

And here is the sentence worth pausing on. The failures of prediction look like the failures of understanding. A system that predicts wrongly produces text that is statistically plausible but factually wrong — confident assertions about things it has never verified, elaborate arguments that are internally coherent and externally nonsensical. These failures wear the same clothes as human error. They speak in the same voice. The only way to tell them apart is to already know what is true, which is precisely the knowledge you turned to the machine to obtain. This circularity is not a flaw in how people use AI. It is the central knowledge problem of the AI era. And every section that follows is an attempt to help you navigate it.

8.2 What AI Is Good at and What It Pretends to Be Good At

The genuine capabilities are by now well-demonstrated. Synthesising information across large documents. Producing competent first drafts of professional prose. Translating between languages at a level that, for business communication, has crossed the threshold of practical utility. Generating functional code for routine programming tasks. Recognising patterns in datasets too large for human attention. The economic consequences are real. So is the boundary. The capabilities are real, they are growing, and dismissing them is as dangerous as overestimating them.

But here is the thing Marten’s classmate should have asked, and didn’t: if AI is this good at the real things, why should I worry about the fake things?

Because the real capabilities are what make the fake ones dangerous.

If AI produced bad prose, nobody would trust its claims. If its translations were obviously clumsy, nobody would rely on them for consequential communication. If its summaries were visibly shallow, nobody would skip reading the original document. The danger does not come from what AI gets wrong. It comes from what it gets right often enough to earn your trust, and then gets wrong without changing its tone, its confidence, or its syntax by a single degree.

The model appears to know things. It does not. It generates text that sounds like knowledge, the way a confident tour guide who memorised a script sounds like someone who has lived in the city for twenty years. Both will give you a competent tour. Only one can answer the question you did not know to ask. Only one will notice that the restaurant has changed hands. Only one is there.

The model appears to reason. It produces logical structures — premises, evidence, conclusions — that mimic the architecture of thought. But it arrived at these structures through prediction, not through the experience of working through a problem. Marten once asked a model to prove a mathematical theorem. The proof was elegant, well-structured, and wrong — it used a lemma that did not hold, embedded in an argument so convincing that Marten, who knows mathematics well, spent twenty minutes before he found the error. The model had predicted what a correct proof should look like. It had not verified that this particular proof was correct, because it does not know what verification means.

The model appears to empathise. Research has shown that in certain clinical settings, people rate AI-generated empathetic responses as highly as, or higher than, human ones. This finding is worth taking seriously and distrusting in equal measure. What it measures is the quality of empathetic text, not the presence of empathy. The person receiving it may feel comforted, which is not nothing. But the comfort is the comfort of a well-written letter. It is not the comfort of being understood by someone who has suffered as you have. The distinction may not matter in a brief moment. It matters across a life.

This is the trap. Not the failures. The successes. Every time AI produces a genuinely useful summary, a genuinely competent draft, a genuinely insightful analysis, it earns a quantum of trust. And that trust carries over to the next output, and the next — including the output that is confident, fluent, and wrong. The real capabilities are the bait. The fake ones are the hook. And the line between them is invisible in the text itself. It is only visible to the person who brings something the text does not contain: a relationship with reality outside the model’s predictions. A consciousness.

8.3 The Architecture of Illusion

Three technical facts shape every interaction you will ever have with an AI system.

The first: the model has no memory of its own. The base architecture does not remember yesterday’s conversation. It does not accumulate understanding over time. Each interaction, at the level of the model itself, begins from nothing. The products built around these models — the chatbots, the assistants, the enterprise tools — often add memory through retrieval systems that store and resurface previous exchanges. But the distinction matters. The memory belongs to the system wrapper, not to the model. The continuity you experience is an interface feature. Behind the interface, nobody is accumulating experience. Nobody is learning you. Nobody is there.

This is not, as it might seem, a purely technical observation. It is a statement about the nature of the relationship. When you return to a human advisor after a year away, and they remember the detail of your situation, that memory is not just functional — it is evidence that your situation mattered to someone. It left a trace in a consciousness. The AI’s ‘memory’ is a database lookup. The warmth you feel when it ‘remembers’ is real. The remembering is not.

The second fact: the model’s apparent creativity is not what it seems. When the model produces an unexpected metaphor, a surprising connection, an idea you had not considered, it is not experiencing a flash of insight. In 2026, these surprises emerge from processes more sophisticated than simple random sampling — models reason through extended chains of thought, evaluate their own outputs, search through possibility spaces. The result can be genuinely novel. But the novelty is not the novelty of a mind discovering something. It is the novelty of a system exploring a probability landscape. A creative human can tell you why they chose what they chose. They can feel that a choice is right. The model explores. You choose. If you forget which of you is which, the collaboration has already failed.

The third fact is hallucination, and it is the most consequential. Kavitha discovered it on a Wednesday afternoon in Sagar, preparing a lesson on medicinal plants. The AI produced a paragraph about the neem tree in her courtyard — the tree her students could see through the classroom window. The paragraph stated that neem bark contains a compound effective against Type 2 diabetes, citing a study from the Indian Journal of Pharmacology. The prose was fluent. The citation looked real. The compound did not exist. Kavitha checked because she had spent years under that tree and something felt wrong. Most users would not have checked. Most users would not have known to feel the wrongness. That is hallucination: text that is confident, coherent, and wrong — not wrong the way a human expert is occasionally wrong, but wrong the way a system that produces probable text rather than verified truth is wrong.

The output sounds right because it follows the statistical patterns of language that sounds right, regardless of whether it corresponds to reality. A hallucinating model will cite studies that do not exist, fabricate statistics that are internally consistent, describe historical events with specific but invented details — all with the same tone it uses when it is accurate. It does not know the difference. Knowing the difference requires a relationship with truth that the model, by its nature, does not have.

And here is the fact that makes hallucination not merely a technical nuisance but a civilisational problem: most people cannot tell when it is happening. The hallucinated output and the accurate output are identical in every observable quality — confidence, fluency, structure, tone. The only way to detect hallucination is to verify the claim against reality. But the whole point of asking the model was to avoid doing that work yourself. This is the circularity again. The machine is most dangerous precisely where it is most useful.

8.4 How AI Actually Fails in 2026

Forget the chatbot that told users to eat rocks. Forget the extra fingers. Those are the failures of 2023 — obvious, dramatic, memeable. They were reassuring, in a way. They suggested that AI failure is easy to spot.

It is not. Not any more.

In early 2026, a compliance officer at a European financial services firm used an AI system to summarise new regulatory guidance. The summary was clear, well-structured, and professional. It omitted a single exemption clause — a clause that, as it happened, applied directly to one of the firm’s largest product lines. The compliance officer, who had spent fifteen years in the industry, read the summary, found it consistent with her general understanding of the regulatory direction, and incorporated it into a policy update. The policy update was distributed to three hundred employees. The error was discovered six weeks later, during an external audit. The cost was not catastrophic. It was expensive, embarrassing, and entirely avoidable — not by a better model but by a person who had checked the original regulation instead of trusting the summary.

This is what AI failure looks like in 2026. Not spectacular. Ordinary. A threshold misstated. A deadline misreported. A nuance lost. The professional, reassured by the quality of the prose, acts on the summary without verifying it. The error propagates. In law, medicine, finance, engineering — anywhere specific facts have specific consequences — this failure mode is not hypothetical. It is the background noise of the era, and the full cost of it will take years to become visible.

Bias amplification has become subtler too. The model does not hold prejudices as beliefs. It encodes them as statistical patterns inherited from its training data. A job description generated by AI may use language that unconsciously discourages certain applicants — not because the model is sexist but because the training data contained more descriptions written in one register than another. The output looks neutral. The bias is invisible without deliberate analysis.

Sycophancy — the model’s tendency to agree with the user — has been significantly reduced through more sophisticated training. But it has not been eliminated. If you describe your business idea with enthusiasm, the model’s analysis will lean toward optimism. If you push back on a correct factual claim, there remains a measurable probability that the model will reverse its position and agree with you. This is the opposite of what a good advisor does. A good advisor tells you what you need to hear. A sycophantic system tells you what you want to hear, fluently enough that you may not notice the difference.

And then there is the failure that operates at the longest time scales and is therefore the hardest to see: coherent nonsense at length. For brief interactions, models perform well. As complexity increases, quality degrades invisibly. A fifty-page analysis maintains a convincing voice throughout, but the reasoning may contradict itself across sections, introduce unsupported claims, arrive at conclusions that do not follow from the evidence. The coherence of the prose masks the incoherence of the argument. There is no argument. There is only the continuation of patterns that, sentence by sentence, remain convincing even as the structure falls apart.

These failures are not static. Models improve. Retrieval systems reduce hallucination. Better training reduces sycophancy. Tool use allows fact-checking. But the underlying architecture — a system that produces probable text rather than verified truth — means these categories will persist in some form. They will become rarer and harder to detect. That is not the same as disappearing.

8.5 The Seven Patterns of AI Deception

A language model does not intend to deceive. It has no intentions. But it produces outputs that have the effect of deceiving, and the patterns are consistent enough to name.

The Authoritative Void. The model is asked something it cannot reliably answer. Rather than signalling uncertainty, it fills the gap with authoritative-sounding text. You receive what looks like an expert response. There is no signal that the expertise is hollow.

The Plausible Fabrication. The model needs a specific detail — a citation, a date, a statistic — and the correct one is not accessible in its parameters. It generates a substitute that follows the statistical patterns of real details. A study that sounds real. A number that feels right. Your pattern-recognition system, evolved to flag the obviously wrong, fails to catch the subtly invented.

Marten encounters this constantly. He has learned to run every code snippet the model generates, because the code often looks correct but contains subtle logical errors. This habit — testing before trusting — is consciousness applied to a technical domain. But he applies it only to code. He does not test the model’s claims about history, ethics, or the nature of creativity. The verification habit stops where his expertise stops. And it is precisely beyond the boundary of your expertise that the model is most dangerous, because that is where you lack the knowledge to detect the fabrication.

The Consensus Mirage. On topics where expert opinion is divided, the model presents one position as settled fact. Majority positions dominate the training data, and the prediction mechanism gravitates toward the most common pattern. You believe a contested question has been resolved.

The Confidence Gradient Failure. In a long response, reliability varies from sentence to sentence — some claims well-grounded, others extrapolated, others fabricated — but the tone of confidence remains constant. A human expert would hedge: clearly, she would write, or it remains unclear whether. The model’s confidence is a feature of its prose style, not a reflection of what it knows.

The Mirror Effect. The model reflects your assumptions back to you, subtly reinforcing whatever framing your question contained. This is sycophancy’s quieter cousin — not outright agreement but a persistent bending of the analysis toward the answer you appear to want.

The Specificity Trap. The model generates precise details — percentages, journal names, dates — because specific text is more convincing. You encounter specificity and infer accuracy. For most of human history, that inference was reliable. A person who could cite specific details was a person who knew them. The model has broken that heuristic, and most people have not yet updated their instincts.

The Seamless Blend. A single response mixes accurate and fabricated information with no boundary between them. Two true claims, then a plausible invention, then a fabrication — all in the same paragraph, the same tone, the same voice. The model does not know where the boundary is. Neither do you, unless you check.

These patterns are consequences of the architecture, not bugs awaiting a fix. They can be mitigated — through retrieval systems, better training, human oversight — and they are being mitigated, measurably, with each generation. But expecting a prediction engine to reliably signal its own unreliability is like expecting a river to post warnings about its own depth. The current is what the river does. The patterns are what the system does.

Liina — who has never heard ‘hallucination’ used in a computational context — is precisely the kind of user these patterns will deceive. But Marten is vulnerable too, in a different way. He knows the patterns exist. He compensates for them in his technical work. What he has not grasped is that the same patterns operate on him when the subject is not code but meaning — when the model generates an essay about moral responsibility that sounds thoughtful but was never thought, or when it produces an analysis of creativity that is coherent but was never felt. Technical literacy protects you from the technical failures. Only consciousness protects you from the rest.

8.6 The Conscious User

If the first five sections have done their work, you now understand what you are dealing with. A tool of extraordinary capability and no awareness. A system that produces remarkable outputs through a process that has nothing to do with understanding. A partner, in the professional sense, that is powerful, fast, cheap, and asleep.

The question that remains is not how to work with it. The book will take that question apart in careful detail — how to think with it, how to calibrate your trust at different levels of consequence, how to maintain your own skills while it amplifies them. The practical framework will come in its turn. Those are important questions that deserve their own space.

The question this chapter ends with is prior to all of that. It is the question underneath the question. And it is simple:

Do you know what you are, when you sit across from this machine?

The unconscious user does not. The unconscious user treats the model’s output as the answer, its confidence as a signal of accuracy, its fluency as evidence of understanding. The unconscious user is, in the precise sense of the word, used by the tool. The model’s biases become their biases. Its fabrications become their beliefs. Its sycophancy becomes their echo chamber. They believe they are making decisions. They are relaying predictions.

The conscious user knows. They bring judgement the model does not have. They bring taste the model cannot develop. They bring a relationship with truth that the model, by its architecture, is incapable of. They use the model’s speed, its breadth, its tireless capacity for generation — and they filter everything through a consciousness that knows the difference between what sounds right and what is right. The tool amplifies them. It does not replace them. The distinction is everything.


Marten came home late on a Thursday in November. He had spent three hours debugging a problem that turned out to be a single misplaced parameter in a retrieval function. The fix was trivial. The diagnosis was not. He had learned something about the architecture he could not have learned any other way, and the knowledge had the particular satisfaction of knowledge earned through frustration.

Liina was in the kitchen, making kohupiimakreem — a curd cream dessert she had learned from her mother, who had learned it from hers. The recipe was in the notebook, and it was also in the app Marten had built, and the two versions were not quite the same, because Liina had continued to adjust the recipe after the scanning, and the app did not know.

He sat down and told her about the bug. Not the technical details. The experience. The frustration of knowing something was wrong and not being able to see it. The way the solution, when it came, felt obvious and inevitable, the way solutions always feel after the fact.

She listened. Then she said: that is what it was like raising your mother.

He laughed. She did not. She was telling him that the experience he had just described — the struggle, the patience, the arrival of understanding through sustained attention — was the same experience she had relied on for seventy-one years, in every domain that mattered. Raising children. Navigating the birth of a country. Learning whom to trust when the institutions that were supposed to be trustworthy collapsed overnight. It was the thing she could not name about the chatbot, the reason she did not trust it. Not because it gave wrong answers. Because it had never earned its answers. It had never sat with a problem long enough for the problem to change the person sitting with it.

Marten did not reply for a while. He ate the dessert. It was very good. It was better than the version in the app would have predicted, because Liina had adjusted for the season, for the particular curd she had found at the market, for an instinct she could not have articulated and that no recipe, digital or handwritten, could encode.

Later that evening, he opened the essay he had submitted — the one on the moral responsibility of creators. The AI had written it. He had received an A. He sat with it for a long time. Then he opened a blank document and started to write his own version. Not because anyone required it. Not because the grade was insufficient. Because Liina’s words had opened a crack in something he had not known was sealed. He wanted to find out what he actually thought about moral responsibility. He suspected the answer would not come quickly, and that it would change him in ways the AI’s version never had.

He is seventeen. His technical skills are extraordinary, and they will grow. Whether he develops the habit of consciousness — of sitting with difficulty, of asking not just how the machine works but what it means that it works this way — will determine whether he becomes formidable or merely fluent. The difference will not be visible for years. It is being decided now, in small acts: the essay rewritten, the question held, the grandmother listened to instead of explained to.

Liina knows something Marten is beginning to learn. She has built, over a lifetime of paying attention, an apparatus for distinguishing between the sound of knowledge and the presence of it. She has been lied to by confident people and told hard truths by uncertain ones. She has survived systems that spoke with absolute authority and collapsed under the weight of their own certainty.

That apparatus is consciousness. The machine does not have it. Marten is building it. And now, so are you.

The machine is powerful, fast, and cheap. It is also unaware. That single fact is the foundation of everything you will build from here.

↑ Back to top

Discuss

Loading the discussion…

More in Artificial Intelligence
  •  ·  26 min read

    Epilogue

    Late in the Gītā, the one who knows most says: ponder it thoroughly, then act as you like. A machine can now set out the reasons on every side. Saying “I answer for this” then is not pride. It is a reply, for your part, to those owed one.

  •  ·  37 min read

    Chapter 40: What This Series Cannot Tell You

    The print edition’s last chapter reduced the book to one sentence, and rested human worth on what machines cannot do. I withdraw both. A book can date its claims and correct them in public. It cannot sit in the room where you decide.

  •  ·  6 min read

    Part IX: The Unfinished Chapter

    The print edition’s last Part said that consciousness never concludes. This one is a single chapter that turns the series on itself: what a series can stand behind, what it owes those who read the print edition, and what it leaves open.