NP Nithin Palal

Chapter 17: Intelligence-Native Products

Part IV: Conscious Creation

· Updated 15 November 2026 · 39 min read · Discuss · Plain text

AI-Proof: What Machines Can Never Be — Volume II: Application, Part IV: Conscious Creation, Chapter 17: Intelligence-Native Products

Part IV asks what deserves to be built. Chapter 14 separated surviving from deserving, Chapter 15 asked what a company can call its own when its rivals use the same machines, and Chapter 16 asked how deep a machine should go into the work. This chapter asks what a product that learns from its users does to them, and who must find out.

Priya’s platform caught as many bugs as ever. The developers had stopped reading its findings. A product that learns from people also teaches them, and whether it deserves to be built depends on who finds out what it teaches.

In Brief For readers who build, run or rely on products that change as people use them. Chapter 17 is about products that learn from the people who use them, and so teach them. A test lead in Bengaluru finds her platform teaching a team to stop reading; a language app in Seoul succeeds on every number it measures while its learners lose the nerve to speak; a lender in Dhaka builds screens that decide what its officers see first. Norbert Wiener on the purpose put into a machine, the Bhagavad Gītā on the dark act begun without looking, Hans Jonas on fear and foresight, the Kaṭha’s good and pleasant, and studies of how people judge beside machines help answer who must find out what a product does to its users, and who has the right to decide what it makes of them.

A product that learns from the people who use it also shapes them; it deserves to be built only if someone who can change it will find out what it does to them, and must answer for it.

Opening: Four Seconds a Finding

The log ran on, screen after screen. A finding appeared. A developer marked it acknowledged. Four seconds, or a little less.

Four seconds was the shortest time the screen allowed before the button would work. For six weeks the mobile team had been pressing it almost the moment it lit up.

It was a Wednesday in July 2025. That morning Deepak, the director of platform engineering, had sent Priya Chandran a message of two words: Check mobile.

Priya leads the testing team of twelve at a software company in Koramangala, in Bengaluru. Since March her team had been building a new kind of testing platform, and since June every product team in the company had used it. It learned from every cycle of tests. It moved its attention towards the parts of the code where defects had gathered before. It gave each team its own view. The payments team, which works under banking rules, saw cautious thresholds and a full record of every check. The mobile team, which shipped a new version of the app every week, saw fast results ranked by trend. The infrastructure team, whose failures were rare and severe, saw long histories.

She opened the mobile team’s figures. The first number was the defect escape rate: the share of defects found by customers rather than by testing. It had risen by about a third in six weeks. She looked for the platform’s mistake and did not find one. It was catching as many defects as before, and it had missed nothing it would have caught in May. Then she opened the review log, and the four seconds were there.

The machine had done its part. The people had stopped reading. Each click was a small signature of the kind Chapter 1 described: a name on a check that nobody had made.

There was a second thing in the log, and it was worse. The platform learned from the clicks. A finding marked acknowledged counted as a finding confirmed. For six weeks it had been told, many times a day, that its findings were right. It had been learning from a team that was no longer checking it.

Nobody had decided that the mobile team should stop reading. The team shipped every week, and its lead was judged on whether the release went out on time. Four seconds is about what a weekly release leaves for a finding that looks routine. The smooth screen made skipping easy, and the schedule made it sensible. Priya had built the first. The company had built the second. Each developer still had a part to answer for. A click said that a finding had been read, and it had not been. That part was small beside the screen’s and the schedule’s. Owning it does not make the developers answer for the rest.

This book calls products like Priya’s platform intelligence-native: they could not exist without machine learning. In such a product the learning makes the process work; take it away and there is no product. The word native claims more than it should. In many such products the intelligence is rented from a model someone else owns. Renting the model does not rent out the answering. Whoever builds it into a product answers for what the product does to the people who use it. Still, the name points at something real. Such a product learns from the people who use it. And whatever learns from people also teaches them. Priya had built a platform to learn from the teams. In six weeks it had taught one of them to stop reading.

So the chapter asks: When a product learns from the people who use it and shapes them in return, who must find out what it is doing to them, and who has the right to decide what it makes of them?

The print edition asked a larger question, “does this product deserve to exist?”, and settled it in one line: “The answer depends on whether someone was conscious during the design.” That line is withdrawn. Everyone in this chapter was conscious throughout: Priya, the team in Seoul, the lender’s managers in Dhaka. What they lacked was a way to see what their products did to people, and in some cases the power or the will to act on what they saw.

The larger question still needs sharpening. Deserve to exist is the phrase in the title of this Part, and Chapter 14 separated it from surviving. A product deserves nothing, in the way a hammer deserves nothing. Desert belongs to acts, and to the people who do them. To ask whether a product deserves to exist is to ask whether its makers can answer for building it and keeping it running. They must be able to explain it, undo it where it can be undone, and put right what can be put right. That is the question the chapter tests, product by product.

17.1 Continuous Learning Systems

The print edition said that the defining feature of such a product is that it changes “through continuous interaction with the people who use it”. The product you use on Tuesday, it said, is not the product you used on Monday. “It is not a photograph of a capability, frozen at the moment of release. It is a living system.”

That needs correcting before it can be used. As of 2026 a deployed language model’s weights, the numbers it learned in training, do not change while people talk to it; Chapter 8 called such a model a photograph. What changes is the product built around it, and, between versions, the model itself. It changes what it remembers about each user, in notes it was built to keep. It changes what it retrieves, such as the latest records it is allowed to fetch. And it changes what it is retrained on: systems that rank what you see, recommend what you buy or flag a payment as fraud are often retrained on newer data. Some ranking systems are updated within minutes from what their users click. Even then, what they learn from is someone’s design. So the useful questions are where the change happens and who controls it. Priya’s platform changed in all three ways, and each was someone’s choice. Her team chose to keep a profile of each team and to fetch each team’s defect history before ranking its findings. It also chose to retrain the platform each month on the month’s test results. Nobody had asked what should count as a confirmed finding. A field on a screen, built by her own team, had settled it by default.

So “living system” is the wrong picture. A living thing has its own ends. A learning product has the ends its makers gave it, pursued through data its users supply. That description is less romantic and more useful, because it says where to look when something goes wrong.

Engineers have a name for one common way it goes wrong. In 2015 a group of engineers at Google described the hidden costs of running machine learning in real products. “One of the key features of live ML systems,” they wrote, “is that they often end up influencing their own behavior if they update over time.”1 They called the simplest case a direct feedback loop: a model that helps choose, by its own outputs, the data it will learn from next. Priya’s acknowledgements were exactly that. The platform’s findings produced clicks, and each acknowledged finding went into the next month’s training as a confirmed defect, so the platform gave more weight to findings like it. Receiving a finding, reviewing it and confirming it are three different events, and the platform had counted the first as the third. It had mistaken a signal of compliance for a signal of correctness. A feedback loop of this kind can make a system more confident while it becomes less checked.

The American mathematician Norbert Wiener saw the moral side of this early. Wiener taught at MIT and founded cybernetics, his name for the study of control and communication in animals and machines. In 1960 he wrote in the journal Science: “If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”2

Wiener was writing about machines that act faster than people can stop them. A learning product is often slow. But it shares the condition he named. By the time anyone has the data to intervene, the product has already acted on many people, many times. Its makers rarely watch the purpose they put into it. They watch numbers that stand for it. And a number can be, in Wiener’s phrase, a colourful imitation of the purpose.

The machine has a side here, and it should be said plainly. The platform’s own log showed what the mobile team was doing. No manager walking the floor would have seen four-second clicks, and nobody on the team would have reported them. A learning product can watch the people who use it more closely than any person could. The danger and the help are one fact. What matters is who reads what it sees, and what they are allowed to do about it.

The print edition said that building a system that learns is an engineering problem, and deciding what it should learn from is a judgement problem. It added that “only consciousness could have asked where the engine was headed”. The first half stands. The second is withdrawn. A machine in 2026 can be asked where a system is headed, and will often give a fair reply. Intelligence, however good, does not confer the right to decide. Choosing what a product learns from is choosing what it will make of the people it learns from. That is a decision about other people, and it belongs to people who can be asked why.

17.2 Feedback Loops: What a Product Can See

The print edition compared a product’s measures to a person’s senses, which feed the “conscious mind” that “constructs meaning”, and it added that no product will ever have consciousness. The comparison hides the most important fact. Nobody chose a person’s senses. A product’s senses are whatever its makers chose to count.

The company in this section is a composite. No single firm is meant, and its numbers are rounded on purpose. The pattern is real.

Through 2023 a small team in Seoul built an app to teach Korean to adults. It launched early in 2024. It listened to learners speak. It noted where they paused, which grammar they used and which they avoided, and it built exercises aimed at the gap. The team read a long pause before a hard verb ending as fear, and designed gentle exercises to coax the learner across it. That reading was a guess. A pause can mean fear. It can also mean not knowing the word, or a noisy bus.

Within a year the app had hundreds of thousands of learners. The team’s screens showed exercises completed, errors falling in each session, learners coming back, and time in the app rising. By every number it measured, the app was winning.

In the first half of 2025 the company paid a language teacher to run a small follow-up. She invited volunteers who had used the app for at least three months to talk with a stranger for ten minutes, with no app and no script. She did the same with learners who had spent similar months with a human tutor. The heavy users of the app made fewer grammar mistakes. They also stopped more often, abandoned sentences sooner, and asked the stranger fewer questions. The app had made them good at avoiding errors in exercises. It had not made them able to bear the errors that real talk requires.

The comparison was weak, and the teacher said so. Learners who can pay a tutor are not like learners who cannot. Volunteers are not like everyone else. Ten minutes with a stranger is one test of a language, not all of them. The company could not prove that its app had made anyone worse. It could see that its numbers and its learners’ speaking had come apart. The follow-up had tested the one thing the app could not arrange: what learners could do outside it. A product is judged by what people can do outside the conditions it controls.

Wiener’s phrase fits. Falling errors per session was a colourful imitation of the purpose, which was learning to speak. A related trap has a name, Goodhart’s law: a measure that becomes a target stops being a good measure.3 The Seoul app’s trouble began earlier. Its measure had never been the purpose. It was only the part of the purpose the app could see.

What a product can see comes down to four questions. Did it do what it was built to do? Do people use it the way its makers intended? What happens to them after they close it? And are they becoming more able to do what they came for? The Seoul team could see the first two. The third and fourth lay outside the app: at a pharmacy counter, in a meeting at work, on the telephone to a landlord.

The fourth question is easy to put as a choice between more capable and more dependent. That is the wrong contrast. Everyone depends on tools. Chapter 12 found that the fault is depending without knowing what you depend on, and without anyone who could tell you when it fails. So the honest fourth question is narrower. Do the people the product shapes know what it does for them? And could they do the thing they came for, on the day it is not there?

The machine has a side here too. For drilling grammar the app was better than most tutors. It was patient at midnight and cost a fraction of a lesson. The follow-up compared it with a tutor, but for most of its learners the real alternative was not a tutor. It was nothing. An app that teaches less than a tutor and more than nothing may still be worth building, for the people who have nothing else. Whether this one was depends on what it took from them that nothing would not have taken.

The pattern was already known. A trial with about a thousand secondary-school students in Türkiye had found it. Those given an AI that supplied solutions did better in practice and worse in the exam taken without it. Those given one that offered only hints did not pay that price.4 Chapter 13 saw the same thing from a teacher’s desk in Sagar, in Madhya Pradesh. There Kavitha Sharma’s student Rohit learned to predict the platform’s next question, and she learned to read the sign. The Seoul app shows it from the builder’s desk, which is where it can be changed.

There is a harder problem. To learn the third and fourth, a product must follow people out of the product. It must know what happened at the pharmacy counter. An app that recorded its learners’ lives in order to protect them would harm them in a new way. Watching more is not the cure for not seeing.

The decent route is slower. Ask people, with their consent, and accept a smaller and rougher result. Pay a teacher to run a test with volunteers, as the Seoul company did. Ask learners every few months what they can do now that they could not do before, and what they can no longer do without the app. Let researchers outside the company look. Asking gives the people a product shapes a say in what counts as working. That is a reason to prefer it even where watching would be legal.

17.3 Adaptive Personalisation

Priya’s first fix for the four seconds was wrong.

She spent that weekend building a mandatory review. Before a developer could dismiss a finding, a box now asked for a short assessment in their own words. She put it live for the mobile team on the Monday. She could change any team’s screen without asking anyone, and nobody in the company had thought that strange.

By the Wednesday three developers had complained to their manager. The team’s lead wrote to Priya, politely and with strain, asking whether the platform was now meant to slow them down. The escape rate did not move. In the review box the developers typed ok, fine and checked, and pressed enter. They had found the shortest path through the friction, as anyone judged on speed will.

Two things had gone wrong, and both were hers. She had built a screen so smooth that it made skipping easy, and then a fix so blunt that it made checking a ritual. And she had imposed the fix on a team whose hours it spent, without asking them.

She switched the box off and went to the floor below, where the mobile team sat. She sat with the team’s lead and two of the developers for an hour. One of them said the thing that changed the design. “Most of your findings are obvious,” he said. “You made us slow everywhere. Slow us down where your machine is guessing.”

What they built together over the next two weeks, Priya called selective difficulty. Where the platform was confident about a finding, it showed its own assessment at once. Where its confidence was in the middle, it asked the developer for their own reading first, and only then showed its own. Those were the cases where a person’s reading mattered most. By early August 2025 it was running for the mobile team. Two limits went into it from the start. A finding the platform was sure of could still be wrong, and its sureness drifts on new code it has not seen. So each week a small sample of confident findings still went to a person. And the friction followed what a miss would cost, not only the platform’s doubt. A confident finding in the code that handled logins or payments still waited for a developer’s own reading. The design also changed what the platform could learn from. A developer’s own reading, given before the platform’s, says something about the finding. A click after four seconds says only that the screen was seen. So from August 2025, when the platform retrained on the mobile team’s results, only a developer’s own reading, given before the platform’s, counted as a confirmed finding.

The power that let Priya ship this after listening had let her ship the box without listening. Nothing in the company required the second hour on the floor. Chapter 16 found that a tool can feel “ours” because the people who use it were asked, and that holds between one team and another. That she asked the second time was her choice, not a rule.

The idea was not new to Priya. Since the spring she had formed her own view of a problem before asking the machine. A month later, in Sagar, Kavitha would build the same habit into her notebook. What was new was building it into other people’s work. A habit you choose for yourself is a practice. The same habit built into someone else’s screen is a rule, made by one person about another person’s hours.

The evidence on such rules is mixed, and worth having. In 2021 Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos, two of them at Harvard, tested a kind of design. It asks a person to commit to a judgement before seeing the machine’s suggestion. They called this cognitive forcing. About two hundred people online chose lower-carbohydrate swaps for ingredients in meals, helped by a simulated AI that was sometimes wrong. The forcing designs did more than explanations to stop people following the machine when it was wrong. But “people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most.” And the designs helped most the people who already enjoyed effortful thinking.5 It was a short online task, not a software team over months. Still, its three findings are the honest price of selective difficulty. It can work. People will dislike it. And it may help most the people who needed it least.

In the weeks that followed, the mobile team’s escape rate fell. Priya did not take the fall as proof. A figure that rises for six weeks often falls again by itself, and she could not separate her design from everything else that changed that month. A decision is judged by what was known, not by how it turned out. The design was sound on what she knew. Whether it caused the fall, she could not say. Nor had the design touched the other cause. The weekly release that made skipping sensible was the company’s to change, not hers.

This is the tension the print edition named, and it drew the line in the wrong place. It said that personalisation either serves or exploits. A learning system that meets a student’s confusion with a different explanation serves; the same system “simply lowering the difficulty” exploits. But lowering the difficulty is often exactly right. Good teachers do it every day, and raise it again the next. The difference between serving and exploiting lies elsewhere. It lies in whose end the shaping serves, and whether the person shaped can see it and change it.

The Kaṭha Upaniṣad has the oldest name for it. Death tells the boy Naciketas that the good, śreyas, and the pleasant, preyas, are two different things, and that whoever chooses the pleasant misses his end.6 The text puts the choosing in the person. A product tuned to what its users enjoy chooses the pleasant on their behalf, and serves its own numbers while it does. The Seoul app’s completed exercises were the pleasant; the conversation at the counter was the good. But a product that forces the good on people takes the choice from them as well. Priya’s review box tried that, and it failed. The Kaṭha is about liberation, and its boy chooses for himself. I borrow only its line between the good and the pleasant, and its point that the choosing belongs to the one who lives with the result.

Between choosing the pleasant for people and forcing the good on them lies a third way, which respects the person. A default is always someone’s decision. Every learning product is a set of defaults that move: what to show first, what to make easy, what to make hard. Some shaping cannot be avoided. The fair version says what the setting is, who set it and why, and lets the person shaped change it where the choice is theirs. For a learner on a phone, most of the choice is theirs. For a developer at work it is shared. A company may decide how work that carries its name is checked. But the people whose hours a check spends should help design it, and the person who imposes it must answer for it. Priya learned that after the box failed, not before it.

17.4 Predictive Architectures

Prediction is not new. Insurers and weather forecasters lived by it long before computers. What is new is its price. A product can now predict, for every user and every hour, what once needed an analyst and a week.

Priya’s platform predicted, after every change to the code, how likely each module was to contain a serious defect. The predictions were useful. They let teams spend their testing where trouble was likeliest. But they arrived as single numbers, such as seventy-three per cent, with more precision than the model had.

In August 2025 Priya redesigned the screen. Beside each prediction it now showed the signals behind it, how strong each signal was, and where the model’s past predictions had been weakest. She called these confidence surfaces. Her technical lead objected. Teams would read instead of act, he said, and a number is clearer than a paragraph.

Priya was sure he was wrong. He was partly right.

In March 1951 an American intelligence estimate said that an attack on Yugoslavia that year, prepared by the Soviet Union’s satellite states, “should be considered a serious possibility”. Sherman Kent was a Yale historian who led the CIA’s Office of National Estimates from 1952 to 1967. He later asked his colleagues on the board that agreed the estimate what odds they had meant. “The low man was thinking of about 20 to 80, the high of 80 to 20.”7 The same words had meant one chance in five to one reader and four chances in five to another. Kent called such phrases words of estimative probability, and he and a colleague proposed tying each to a range of numbers.

So a description is not more honest than a number just because it is longer. “Likely, based on three signals” can hide as much as seventy-three per cent hides, and it hides it differently for each reader. What makes a prediction honest is its record. Of the modules flagged at around seventy per cent, how many really had a serious defect? A number is honest only for a defined event, here a serious defect found within three months of a change. And it stays honest only while its record is rechecked as the code changes. A number with its record beside it can be checked. A word cannot. Priya and her lead settled the screen with the teams who would read it. It shows the number, the record of numbers like it, and the signals, in that order.

There was another reason to show the uncertainty. Researchers have found that people lose confidence in an algorithm faster than in a person after seeing both make the same mistake.8 A screen that shows every prediction with the same calm certainty prepares exactly that fall. The first wrong number will feel like a betrayal, because nothing said it could be wrong.

In September 2025 a developer named Arvind came to Priya’s desk. He had been with the company since 2018 and worked on its oldest code, the parts nobody else wanted to touch. He asked, carefully, whether the platform was tracking his personal rate of introducing defects.

It was not. The predictions were made for modules, not for people. But Arvind had noticed that the numbers on his modules rose after his changes, because old and fragile code is risky to change at all. And since he was nearly the only person who changed those modules, the modules’ figures were, in practice, his. He had begun running extra checks before every change. His work had slowed. Nothing in the platform asked him to do this. The numbers felt like a verdict.

Priya’s first answer was the wrong one. She explained how the predictions were grouped, and that no individual could be named. Arvind thanked her and left. By the next morning she saw two problems with what she had said. It was not quite true, because grouping by module does not hide a person who owns a module alone. And it was not hers to promise. What the company did with such figures in its appraisals was decided by Deepak and the people who ran appraisals, not by the person who built the platform.

She also saw how she could find out whether others felt as Arvind did. The activity logs recorded how often each developer ran checks before a change. She opened them and closed them again. Reading people’s logs to learn whether they feared being watched would have been the very thing they feared. The review log she had read in July was different in purpose. It recorded checks made in the company’s name, and she had read it to learn what her platform was doing to the checking. These logs would have been read to learn what people felt.

She asked instead. At the infrastructure team’s next meeting, with Deepak in the room, she described Arvind’s question without his name. Two other developers said they had wondered the same. Three changes followed. The screen now said, beside every prediction, that it described code and not people. Modules with a single owner appeared only inside larger groups. And Deepak wrote into the team’s appraisal guidance that the platform’s predictions would not be used to judge individuals, because only he could make that promise. A promise on a screen also needs a rule behind it. Deepak wrote one with the team. It says who may read the activity logs and for what, how long they are kept, and who may change it: Deepak, and only after telling the team. In October 2025 the promise had not yet met an appraisal round.

What happened was plain. A careful person came to her desk afraid that the numbers were judging him. The design could not remove the fear by itself, because the fear was about the company, not the screen. Someone with authority over the company’s use of the numbers had to answer it.

17.5 AI-Augmented Interfaces

An interface is the part of a product a person sees and touches: the screen, its order, what it puts first. When an interface learns, its order changes for each person. Someone designed how it changes.

The lender in this section is a composite. It lends small sums to shopkeepers and tailoring workshops in Dhaka, through loan officers who visit them. In 2024 it built a screen that arranged itself around each officer’s way of working: what they opened first, what they stayed on, what they skipped. Over the months it learned to put at the top, ready-made, the analysis each officer was weakest at. A senior officer, who read people well and was less sure of cash flow, saw finished cash-flow forecasts first. A junior officer, careful with figures and new to the city’s markets, saw first a score of each borrower’s standing in the local market, built from records he never saw.

Early in 2025, about six months in, the lender’s managers noticed what had changed. The senior officer had stopped doing her own cash-flow sums. The junior officer relied on the standing score without knowing how it was made. Both were deciding faster. Neither was learning the part of the job they were weakest at. The screen had filled their gaps so well that it took away the practice that might have closed them.

Here the machine has a strong case. On the lender’s own records, the junior officer’s loans made with the score went bad less often than the loans of his first months, when he had judged strangers alone. Fewer bad loans is not the whole measure, because a lender can lower its defaults by turning away people who would have repaid. But on what the lender could see, the score helped. So the problem was not that he relied on it. The fault, as in Seoul, was depending without knowing what he depended on. He could not tell when the score was wrong, and he could not explain it to a borrower who asked.

From the middle of 2025, once a week, each officer’s screen showed them the layout another officer saw. The officers disliked it. They were paid partly by the number of loans they approved, and the slow day cost them. Some things were caught that would have been missed. One sound loan was delayed long enough that the borrower went elsewhere. By 2026 the managers called the result mixed, and kept it.

The borrowers never saw either screen. They could not see the score or contest it, and nobody had designed a way for them to ask why.

Priya met a gentler form of the same question. In the summer of 2025 her team added a way for managers to ask the platform questions in ordinary words. A manager could type: which modules have had the most defects over the last three releases? When a question could be read two ways, the platform did not guess. It asked: “Defects can be counted by number or weighted by how serious they are. The rankings differ. Which do you mean?” That was good design. It is also a fact about machines: as of 2025 they can be built to stop and ask, and Priya chose to build one that did. A screen that guesses gives a manager a result. A screen that asks gives the manager back the part of the question that was theirs.

Smoothness is no enemy. An interface that makes routine work smooth frees attention for the work that needs it. The danger is smoothness in the place where a person’s own judgement was supposed to happen, chosen by someone else, and invisible to the person it shapes.

17.6 The Test: Would This Product Know If It Was Hurting Someone?

The test is one question: would this product know if it was hurting someone? It is the best single test in this Part of the book. But know carries four different things, and the test works only when they are pulled apart.

Before the four comes a condition, and Wiener named it: the purpose put into the machine. Before anyone asks whether a product would know it was hurting someone, someone must be able to say who chose what counts as success, and for whom. A product can detect well, be watched closely, be changed by the right people and be explained in full, and still pursue the wrong end. The Seoul app counted errors avoided, while its learners had come to speak. Priya’s platform counted clicks as confirmations, because nobody had asked what a confirmation was. So this is not a fifth step. Each of the four looks for harm against a purpose, and if the purpose is wrong, all four look in the wrong place.

The first is detecting. Can the product register a sign that something is wrong? As of 2026 it often can, and better than any person could manage. Priya’s platform recorded every four-second click. The Seoul app could have asked its learners each month how their speaking was going. Detecting is something machines do, and what they can detect changes year by year.

The second is looking. Does anyone read the sign? A log nobody opens detects nothing that matters.

The third is being allowed to act. Can the person who reads the sign change the product, even when the change will lower the numbers the company rewards?

The fourth is answering. Must someone explain what happened, undo it where it can be undone, and put right what can be put right?

A machine can do the first. The other three need someone who can act and be asked why, and the fourth is not a machine’s to carry, however well it detects. Chapter 7 set out when a position is full and when it is only a name. The person who must look, act and answer holds such a position, and in most products it is empty.

The print edition said that for most intelligence-native products in 2026 “the honest answer” to its test “is no”, because the feedback was never built to detect harm. Sometimes that is true. But the hardest case is the one where detecting worked.

In September 2021 the Wall Street Journal reported on research inside Facebook about Instagram and teenagers. One slide, from 2019, said: “We make body image issues worse for one in three teen girls.”9 The company, which renamed itself Meta the next month, said the reporting misread its research. The figure, it said, applied to girls who already reported body image issues, as the Journal had also noted, not to all girls. Some of the work “relied on input from only 40 teens”, and it “did not measure causal relationships between Instagram and real-world issues”.10 The company then published the slides with its own notes. Which reading is right is not this chapter’s to settle. What the case shows is narrower. The company’s own researchers had looked. The Journal reported that top executives had reviewed the research, and that it was cited in a 2020 presentation to Mark Zuckerberg. So finding out, and even reading at the top, need not be the missing part. Law has now begun to name who must look. Since August 2023 the EU’s Digital Services Act has required the largest platforms, Instagram among them, to look for such risks every year. The risks include harm to minors and to users’ mental well-being. The platforms must act to reduce them, and pay for independent audits.10 A law can require the looking. What a report cannot show is who could act on the findings against the company’s numbers, and who must answer to the people they were about.

Priya had met the same gap at her own scale. In February 2025 she showed seven senior engineers and the vice-president a figure that was wrong (Chapter 12). Part of the fault was her reading; part was the company’s, which kept its worst failures in a separate system no summary could see. The change she proposed in that review became this chapter’s platform. It carried her lesson with it: the edge of what a system can see is someone’s decision, even when nobody remembers making it.

Two old texts help with the second, third and fourth parts of the test. They agree about the fault and disagree about the remedy.

The Bhagavad Gītā, near its end, sorts actions by the quality of mind they come from. In Kashinath Trimbak Telang’s translation of 1882: “The action is called dark, which is commenced through delusion, without regard to consequences, loss, injury, or strength.”11 The Sanskrit, as I read it, carries an image. The dark act, tāmasa, is begun anapekṣya: without regard, literally without looking towards. What it fails to look at is a list. Anubandha is what follows from the act. (Elsewhere in this book the same word names the matters settled before a work begins.) Kṣaya is loss. Hiṃsā is injury, harm done to others. Pauruṣa is one’s own strength, what one can actually do and bear. The verse describes one person’s act and the state of mind behind it. I borrow only its list of what must be looked at, for a team about to build something that will reach many people.

The German-born philosopher Hans Jonas (1903–1993) wrote The Imperative of Responsibility (1979; in English, 1984). He wrote it for an age in which human power reaches further than human foresight. He proposed a new imperative: “Act so that the effects of your action are compatible with the permanence of genuine human life.”12 He was writing about the survival of humanity and of nature, not about apps. I borrow only his point about the gap between what we can do and what we can foresee. For that gap he proposed what he called a “heuristics of fear”. We learn what matters by imagining its loss: “We know the thing at stake only when we know that it is at stake.” From this he drew a working rule. Where the stakes are large and our forecasts weak, give the bad forecast priority over the good one.12

The two lamps agree that not looking is the fault. They part over what to do when looking is not enough. Jonas lets fear lead where what could be lost is humanity itself, its existence or its essence. There the bad forecast is not merely weighed before hope: such a loss must never be staked at all, and not knowing is itself a reason for restraint. His rule is for stakes of that size; it does not ask us to stop whenever we are unsure. The Gītā asks for a different state of mind. Its next verse calls an agent good who is “unaffected by success or ill-success” (18.26), the evenness the Gītā also praises in its second chapter. Read with 18.25, the good agent looks at what follows, and then acts without letting fear or hope bend the act.

The Seoul app tests both. A fear that stopped whenever it was unsure would have kept a cheap, patient tutor from learners who had no other, and Jonas does not ask for that. Evenness alone would let a team say “we looked”, file the follow-up, and carry on as before. Brought down to the size of an app, which is my use and not his, the fear asks less. It asks that a grave possible loss, here the nerve to speak, be weighed before the hope of growth. And neither text asks the people the act lands on. The Gītā is spoken to a warrior on a battlefield, and Jonas addresses the policy of a whole civilisation. A product’s makers can do something neither text imagined, because a product’s users can be asked. Its makers can look, with consent. They can tell the people it shapes what it is doing to them. And they can give them a hand on the settings where the choice is theirs.

What the Seoul company did, late in 2025, was decided by its founders and investors. They made live, unscripted conversation with a paid partner part of the course by default, told learners why, and let them switch it off. It cost money. Time in the app fell. By the spring of 2026 about a third of paying learners had left. By the autumn nobody could yet say whether the learners who stayed spoke better, because the volunteers who would show it were hard to bring back. The company had chosen to find out. It had not yet found out. On this book’s reading, not Jonas’s, this was the practical middle between a fear that stops and an evenness that carries on. The company took the possible loss seriously, kept the product, told its learners and let them choose.

That is what the test asks for, and it is less than a success story. Nothing in this chapter shows that building the means to find out pays. Sometimes it costs a company a third of its customers. The print edition promised that the companies that build such loops “now will be the companies that survive”. That promise is withdrawn. A company that looks may lose to one that does not. The test is no strategy. It asks whether the makers of a product can answer for what it does to the people it shapes. And answering is necessary, not sufficient. A company can explain a harmful product perfectly, put right each case that reaches it, and keep the product because it pays. That is why the test begins with who chose what counts as success.

So the question stays, and now it can be asked whole. Would this product know if it was hurting someone? Ask first who chose what it counts as success, and for whom. Then ask whether anyone who can change it would find out, in time; whether they would be allowed to act; and whether someone would answer.

What would prove this wrong?

My claim here is that, in the hardest case, detecting harm is not the missing part. What is missing is someone with the power and the duty to act on it, against the numbers a company rewards. The European Union is now testing that claim.

Under the Digital Services Act, as this section says, Instagram must already look. If that duty, on its own, led platforms to change designs that cost them use, I would be wrong about where the gap lies. Looking would have been enough.

The record so far points the other way. On 16 May 2024 the European Commission opened formal proceedings against Meta. It was concerned that the systems of Facebook and Instagram “may stimulate behavioural addictions in children”. Its concern rested partly on its reading of the risk assessment Meta had sent it in September 2023. So the company did the looking, and a regulator acted on it. On 29 April 2026 the Commission gave its preliminary view that Meta had failed to “diligently identify, assess and mitigate” the risks of children under thirteen using its services. A final decision could bring a fine of up to 6 per cent of Meta’s worldwide turnover. The Commission said that its inquiry into other possible breaches continued.10

Where to look: the Commission’s decisions in that case, and Meta’s design changes before and after them. A change made under threat of a fine does not count against me, since a regulator can make the company answer.

The flat in Koramangala was quiet that Saturday evening, 11 October 2025, except for the autorickshaws below, which are never quiet. Priya’s mother had gone to bed. Turing, the cat she had named at twenty-two, when she thought intelligence was the most interesting thing in the world, was asleep on a stack of printed design notes.

On Wednesday the chief product officer had sent her an email with a one-word subject line, Architect. It asked her to lead a new effort to carry the platform’s design across every product the company built. She had said yes. She had not yet told her mother, who would ask what an architect does.

The word in the subject line was a job title. In Chapter 9 it had named something else: a way of working, open to anyone on any day, and never a rank. The new post gave her a seat in some of the rooms where the purposes of products were decided. It did not give her the right to decide what other teams’ products were for. And the people those products would shape would not be in the room. Nor did the post promise that looking would be rewarded. Looking would cost the Seoul company a third of its paying learners. What the products were for stayed the CPO’s to decide, with the teams that built them. How figures were used in appraisals stayed with Deepak and the people who ran them. What the platform taught the people who used it, she had built.

Ending: The First Item on the List

Her list was open on her laptop. The first item had not changed since August: the platform could not tell whether the developers who used it were becoming better engineers or only faster ones. She asked every team to build that kind of measure into their products. She had not built it into her own. Below it were newer items. Whether selective difficulty had caused anything. Whether Deepak’s promise to Arvind would hold when the appraisals came. Whether the mobile team still gave their own readings first, or had found a shorter path.

The platform would run its tests again overnight, and next month it would learn from them. If it was teaching the teams something she had not meant it to teach, it would not be called into a meeting to explain. She would. That weight was hers.

She read the first item again. She did not yet know how to build it.

Practice

Follow one person out of the product. Choose one product you build, run or rely on at work: an app, a screen others must use, a score that decides something about someone. Write one line saying what it is for, in the words of the people it is for, not the words of its reports. Then write how anyone would find out if it was making that worse: by what route, how soon, who would read it, and whether that person could change the product. If the honest reply is “nobody” or “the next survey”, write that down. Then ask one person who uses it what it has changed in them: what they can now do, and what they can no longer do without it. Ask with their permission, not by reading their data. Put what they say beside the number the product reports, and see whether the two agree. This takes an hour of your time and some of theirs. If the hour belongs to your employer, ask for it. If raising what you find is not safe where you work, write down what stops you. The practice will not mend the product, and it promises you nothing. It tells you whether anyone would know.

Words to keep

  • anapekṣya: without regard; literally, without looking towards (Gītā 18.25).
  • anubandha: what follows from an act; its consequence (Gītā 18.25). Chapter 9 used the same word for the matters settled before a work begins.
  • cognitive forcing: a design that asks a person to commit to a judgement before seeing the machine’s answer.
  • default: a setting that holds unless someone changes it; always someone’s decision (Chapter 5).
  • defect escape rate: the share of a product’s defects found by customers rather than by testing.
  • feedback loop: a system whose outputs help choose the data it learns from next.
  • hiṃsā: injury; harm done to others.
  • intelligence-native: of a product that could not exist without machine learning.
  • pauruṣa: one’s own strength; what one can actually do and bear.
  • śreyas and preyas: the good and the pleasant (Kaṭha Upaniṣad).
  • tāmasa: of the quality of tamas, dullness and delusion; Telang’s “dark”.
  • words of estimative probability: Sherman Kent’s name for words such as “likely” used to state odds.

Sources

  1. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo and Dan Dennison (Google), “Hidden Technical Debt in Machine Learning Systems,” Advances in Neural Information Processing Systems 28 (2015), section on feedback loops, p. 4: “Direct Feedback Loops”; “Hidden Feedback Loops”.
  2. Norbert Wiener, “Some Moral and Technical Consequences of Automation,” Science 131, no. 3410 (6 May 1960): 1355–1358, at 1358.
  3. C. A. E. Goodhart, “Problems of Monetary Management: The U.K. Experience” (1975); the usual statement of the law is Marilyn Strathern’s, in “‘Improving ratings’: audit in the British University system,” European Review 5, no. 3 (1997): 305–321, at 308. See Chapter 9.
  4. Hamsa Bastani et al., “Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics,” Proceedings of the National Academy of Sciences 122 (2025): e2422633122; see Chapter 4.
  5. Zana Buçinca, Maja Barbara Malaya and Krzysztof Z. Gajos, “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making,” Proceedings of the ACM on Human-Computer Interaction 5, CSCW1, article 188 (April 2021). An online experiment; 199 participants analysed; meal-ingredient decisions with a simulated AI. Participants higher in “need for cognition” benefited more.
  6. Kaṭha Upaniṣad 2.1 (1.2.1 in the older numbering), trans. F. Max Müller, The Upanishads, Part II, Sacred Books of the East 15 (Oxford: Clarendon Press, 1884); see Chapters 2 and 12.
  7. Sherman Kent, “Words of Estimative Probability,” Studies in Intelligence 8, no. 4 (1964); on NIE 29-51 (March 1951) and his colleagues on the Board of National Estimates. Kent led the CIA’s Office of National Estimates from 1952 to 1967.
  8. Berkeley J. Dietvorst, Joseph P. Simmons and Cade Massey, “Algorithm aversion: People erroneously avoid algorithms after seeing them err,” Journal of Experimental Psychology: General 144, no. 1 (2015): 114–126; see Chapter 11.
  9. Georgia Wells, Jeff Horwitz and Deepa Seetharaman, “Facebook Knows Instagram Is Toxic for Teen Girls, Company Documents Show,” Wall Street Journal, 14 September 2021 (print edition, 15 September 2021, under “Facebook Knows Instagram Is Toxic For Teen Girls, Its Research Shows”). The slide is described as “from 2019”. The report adds: “The research has been reviewed by top Facebook executives, and was cited in a 2020 presentation given to Mr. Zuckerberg”.
  10. Pratiti Raychoudhury, “What Our Research Really Says About Teen Well-Being and Instagram,” Facebook (now Meta) Newsroom, 26 September 2021, updated 29 September 2021 with the two annotated research decks. Facebook renamed its parent company Meta on 28 October 2021. On the law since: Regulation (EU) 2022/2065 (the Digital Services Act), Article 34(1) (providers of very large online platforms “shall diligently identify, analyse and assess any systemic risks”, including under point (d) effects on “the protection of public health and minors and serious negative consequences to the person’s physical and mental well-being”) and Article 37(1) (independent audits “at their own expense and at least once a year”); European Commission, press release IP/23/2413, 25 April 2023, designating Instagram among the first very large online platforms, with four months to comply, and saying that “the protection of minors online and their mental health must be assessed and mitigated”. For “What would prove this wrong?” in 17.6: European Commission, “Commission opens formal proceedings against Meta under the Digital Services Act related to the protection of minors on Facebook and Instagram,” 16 May 2024 (IP/24/2664; digital-strategy.ec.europa.eu): the systems of both platforms, “including their algorithms, may stimulate behavioural addictions in children”, a concern “based on a preliminary analysis of the risk assessment report sent by Meta in September 2023”; European Commission, preliminary findings of 29 April 2026 (EC Representation in Cyprus): Instagram and Facebook failing “to diligently identify, assess and mitigate the risks of minors under 13 years old accessing their services”; a fine, if the findings are confirmed, that “shall in no case exceed 6% of the total worldwide annual turnover of the provider”; the Commission “continues its investigation into other potential breaches” (both checked 4 November 2026).
  11. Bhagavad Gītā 18.25 and 18.26, trans. Kashinath Trimbak Telang, The Bhagavadgîtâ with the Sanatsugâtîya and the Anugîtâ, Sacred Books of the East 8 (Oxford: Clarendon Press, 1882), p. 124. The Sanskrit of 18.25: anubandhaṃ kṣayaṃ hiṃsām anapekṣya ca pauruṣam / mohād ārabhyate karma yat tat tāmasam ucyate. The glosses in the text are mine. The evenness of 2.48 (“equable in success or ill-success”, Telang, pp. 48–49) is quoted in Chapter 13. Chapter 9 uses anubandha in another sense, in the Vedāntasāra’s anubandha-catuṣṭaya (1.5), the four matters settled before a work begins.
  12. Hans Jonas, The Imperative of Responsibility: In Search of an Ethics for the Technological Age, trans. Hans Jonas with David Herr (Chicago: University of Chicago Press, 1984), p. 11 (the imperative), the preface, p. x (the “heuristics of fear”, a term Hauskeller and Coyne cite at p. 21, and the rule giving the bad forecast priority, paraphrased in the text), pp. 26–27 (“We know the thing at stake …”), p. 37 (“Never must the existence or the essence of man as a whole be made a stake in the hazards of action”, as quoted by Dinneen). On restraint: Jonas, Philosophical Essays: From Ancient Creed to Technological Man (Chicago: University of Chicago Press, 1980): “ignorance of the ultimate implications becomes itself a reason for responsible restraint”, said of the “quasi-eschatological” potential of technology (1974 edition, p. 18). Bringing the rule down to the scale of a product, and calling the Seoul company’s choice the practical middle, is this book’s use, not his. Original: Das Prinzip Verantwortung (1979). The wording quoted was checked against Michael Hauskeller and Lewis Coyne, Revue Philosophique de Louvain 117, no. 2 (2019), and Nathan Dinneen, Cosmos and History 10, no. 2 (2014), not against the printed book; Dinneen prints the imperative without “human”, which Hauskeller and Coyne and the text in Jonas’s Philosophical Essays include.

Named people in this chapter are composites, and their conversations and circumstances are constructed; the studies, laws and figures cited are real and sourced.

Revised from the print edition (2026)

  • What the print edition claimed: that whether a product deserves to exist “depends on whether someone was conscious during the design”; that an intelligence-native product “is not a photograph of a capability, frozen at the moment of release. It is a living system”; that “only consciousness could have asked where the engine was headed”; that a product’s measures are like senses feeding “a conscious mind”, and “no product will ever have that”; that “the AI could not have made” the Seoul redesign “because the AI did not know that discomfort and learning are, in language acquisition, the same thing”; that personalisation which lowers difficulty “is exploiting”; that the question is whether users become “more capable or more dependent”; that outcome feedback is merely “often legally constrained” by “tightening privacy regulation”, without asking whether watching people is itself a harm; that “you can only tell the difference from inside a human mind”; that a description of uncertainty is better than a number (“Exactly, Priya said”); that “the honest answer, for most intelligence-native products in 2026, is no”; that “the companies that build the sensory systems now will be the companies that survive”, and that seeing was “good strategy”; that the quality of design decisions “depends entirely on the consciousness of the people making them”; and that the machine cannot “feel the weight of what it might be getting wrong”, and “no system she builds will ever carry it for her”. It also presented the Seoul platform and its study as if real, with “three hundred thousand users”; named a Lagos lender Verdant, which is the name of a real firm; named the engineer Suresh; gave the 2.3 per cent room “eight senior engineers” and called it a “quarterly review”, three times; dated Deepak’s message to October; sent Priya for a solitary walk; ended three stories in a success that proved the thesis (“Learning outcomes improved”; “the developers were paying attention again”; “The discomfort was the point”); and closed sections with “This is what empathy looks like in product design” and “Build for trust. Not for smoothness.”
  • What I now hold: a product that learns from people also teaches them. As of 2026 a deployed language model’s weights do not change in use (Chapter 8’s photograph); what changes is what the product remembers, retrieves and is retrained on, so the questions are where change happens and who controls it. Its commonest failure is a feedback loop in which its outputs choose its next data (Sculley et al.), and the platform’s version mistook a signal of compliance for a signal of correctness. Its measures are what its makers chose to count, and a measure can be “a colorful imitation” of the purpose (Wiener). A product is judged by what people can do outside the conditions it controls. The four questions are kept, the fourth narrowed to whether people know what they depend on and could manage without it; seeing outcomes means asking with consent, not watching. Serving and exploiting differ by whose end the shaping serves and whether the person shaped can see and change it (Kaṭha 2.1; defaults as someone’s decision); a practice built into others’ work is a rule about their hours, with an honest price (Buçinca et al.), and the power to impose it without asking is itself a choice about whose hours may be spent. Friction follows what a miss would cost, not only the machine’s doubt, and confident findings are still sampled. A number with its record is more honest than a word (Kent), but only for a defined event and only while the record is rechecked. A promise about how figures will be used belongs to the one with authority to keep it, and needs a rule behind it. Fewer bad loans is not the whole measure of a lending score, and its borrowers can neither see nor contest it. Before the 17.6 test comes its condition: who chose what counts as success, and for whom (Wiener’s purpose put into the machine); then know splits into detecting, looking, being allowed to act and answering; the hardest case is where detecting worked (the Instagram research shows, not settles, this). Answering is necessary, not sufficient. The Gītā (18.25–26) and Jonas agree that not looking is the fault and differ on fear; Jonas’s rule is for losses to humanity itself, where nothing may be staked and ignorance counsels restraint; scaled to an app, which is my use, it asks that grave losses be weighed, not that we stop whenever unsure; the Seoul company’s choice is, on this book’s reading, the practical middle, and neither text asks the people the act lands on. Products deserve nothing; their makers’ acts are judged. Finding out promises no advantage. Priya answers for what she built; the purposes and the appraisals are other people’s to answer for. Each developer had a small part of the four seconds to answer for; the screen and the schedule had the larger. Renting a model does not rent out the answering. A developer’s own reading is evidence about a finding; a click is evidence only that the screen was seen. Reading a log to learn what a product does to the work differs from reading it to learn what people feel. A law can require the looking.
  • What changed my mind: readers’ objections on the photograph, compliance and correctness, transfer, confident errors, calibration, the borrowers, purpose, Jonas, and Priya’s power over other teams’ screens; the Prologue’s rule against defining people by what machines cannot yet do; Chapter 8’s photograph; Chapter 12’s finding that unknowing dependence is the fault, and its institutional half of the 2.3; Chapter 9’s refusal of “architect” as a rank; the audit’s finding that the Seoul study read as real; Wiener’s purpose put into a machine; Sculley and colleagues on feedback loops; Kent’s estimate of 1951; Buçinca and colleagues on cognitive forcing; the Instagram research and Meta’s reply; the Gītā’s dark act and Jonas’s heuristics of fear, read side by side.
  • What survives: the Seoul app that succeeded on every metric it measured, now labelled a composite; the four layers, as four questions; the four-second clicks and “Check mobile.”; the failed review box, and selective difficulty; confidence surfaces; Arvind’s fear that the numbers were judging him; the officers whose screens filled their gaps; “The system did not guess. It asked.”; the test, “would this product know if it was hurting someone?”; the CPO’s “Architect” email; Turing on the papers, the mother asleep, the autorickshaws; the list whose first item does not change; “That weight was hers.”
  • Section titles in the print edition: 17.2 was “Feedback Loops: A Product’s Sensory System”.

↑ Back to top

Discuss

Loading the discussion…

More in Artificial Intelligence
  •  ·  26 min read

    Epilogue

    Late in the Gītā, the one who knows most says: ponder it thoroughly, then act as you like. A machine can now set out the reasons on every side. Saying “I answer for this” then is not pride. It is a reply, for your part, to those owed one.

  •  ·  37 min read

    Chapter 40: What This Book Cannot Tell You

    The print edition’s last chapter reduced the book to one sentence, and rested human worth on what machines cannot do. I withdraw both. A book can date its claims and correct them in public. It cannot sit in the room where you decide.

  •  ·  6 min read

    Part IX: The Unfinished Chapter

    The print edition’s last Part said that consciousness never concludes. This one is a single chapter that turns the book on itself: what a book can stand behind, what it owes those who read the print edition, and what it leaves open.