AI-Proof: What Machines Can Never Be — Volume II: Application, Part IV: Conscious Creation, Chapter 19: Rethinking Roles and Structure
Part IV looks at what people make with machines. Chapter 17 asked when a product deserves to exist, and Chapter 18 rebuilt the organisation. This chapter turns to the roles inside it: what is left of a role when the machine takes its tasks, and who decides what it becomes.
When machines took the checking, Meera asked her managers what their teams still needed. Then she measured their answers against her own plan. A role is more than its tasks, and asking inside a form you control is still control.
In Brief For readers who manage people, or are managed, as machines take over the tasks their jobs were built on. Chapter 19 asks what is left of a role when a machine takes its tasks, and who decides what it becomes. A quality manager in Hyderabad finds that the machine checks as well as she does; a logistics manager in Rotterdam runs a meeting nobody needs any more; a designer leaves when a studio of sixty becomes twenty; and a drug company finds that its safety model trusts some patients’ reports less than others. The Tirukkuṟaḷ on choosing people for work, Mencius on the farmer who pulled up his corn to help it grow, and a proof about the limits of fairness help answer what a role is made of, and why its future belongs to people with authority, not to the machine.
When a machine takes a role’s tasks, what is left is not a human residue but three places to fill on purpose: someone who answers, someone with the right to decide, and somewhere the next person learns.
Opening: The Form with Four Boxes
The last column of the form had four choices, and Meera Reddy had used all of them.
It was September 2025. The form had come from NovaChem’s head office in Basel, from the global head of human resources. It carried a phrase Meera had not seen in fourteen years at the company: “role architecture review”. Every director was to describe each role that reported to them, list its tasks, and give it a future. There were four futures to choose from: expand, change, at risk and end. The first return was due at the end of October, and the final one at the end of March.
Meera ran operations at the Hyderabad site in Genome Valley, a division of about four hundred people. Among those who reported to her were twelve managers in quality, supply chain, manufacturing and procurement. She filled in the form in a week, in her office on the seventh floor. It was careful work. Each row held a role, its tasks, and the share of those tasks the machines now did. In the margin of her printout, in pencil, she had added something the form did not ask for. Beside the name of Usha Iyer, a quality manager with twenty years in the work, she had written: will struggle.
The form claimed to sort roles. Her pencil had sorted people.
She knew something about plans made for other people by then. Her first plan for the machines had failed, and the rebuild that followed gave the systems a better fit without giving the people a voice (Chapter 16). From April 2025 she had asked each team to redesign its own work. Shalini, her customer service director, had then asked her: “If my team designs this, who answers when it goes wrong: my team, or you?” (Chapter 18). So this time Meera did not send the form. She decided to ask her managers first.
What she did not yet see was that asking can happen inside a frame the asker controls. The questions would be hers. The categories were Basel’s. The pencil note was still in the margin. It took one of her managers four months, and one sentence, to say so.
The machines her division had brought in over two years had done what they were built to do. They had taken over much of the routine checking, scheduling and reporting. The tasks were moving, and Basel’s form asked the right question in boxes too small for it: When a machine takes the tasks that made a role, what is left of the role, and who decides what it becomes?
The print edition answered with a human remainder. When the tasks go, it said, what is left is “what was always underneath: the judgement, the relationships, the capacity to make meaning from ambiguity”. Consciousness, it said, made that remainder valuable. It built a manager who “grows people”, a studio run by directors, and a small team it called “The conscience of the system.” Each of these keeps something true. None of them says what a role is.
The book has already built the parts of a better account. A role is made of three things. The first is its tasks: what a person does all day, such as reading a batch record, running a meeting or making thirty versions of a banner. The second is its positions, in the sense of Chapter 7: who answers for the result, who has the right to decide, and whose life the decision touches. These are three relations, and they need not sit with one person. In this chapter’s company, a quality manager signs for each batch, may stop it, and must explain a bad one if it gets out. A different decision, when a safety signal is investigated, belongs to physicians in Basel. Patients make neither decision, and live with what follows from both. The third part of a role is its rung, in the sense of Chapter 4: the place where the next person learns the work by doing it.
A machine takes tasks. The positions do not go with them. They stay where they were, and either someone holds them or nobody does. If a company lets a machine decide with nobody signing, the position has not moved to the machine. It is empty, and whoever chose to let the machine decide answers for that choice. The rung goes with the tasks, unless someone rebuilds it. So “what does this job become?” is a question with a person behind it. A job does not become anything by itself. Someone redesigns it, or nobody does, and it empties while the title stays.
19.1 AI-Augmented Roles: What Your Job Becomes
Madhavi Krishnamurthy was one of Meera’s twelve. She managed quality assurance for part of the site, with six analysts under her, and she had read batch records for about a decade. A batch record is the full written history of one lot of medicine: every step, every reading, every signature. Madhavi read them the way a musician reads a score. She heard the wrong note before she could name it. Monitoring readings stayed inside their limits but clustered oddly. A report on a fault answered every question a little too neatly. Her analysts brought her the hard cases, and she sat with them at her desk by the window, beside a photograph of her daughter’s bharatanāṭyam recital.
People in the building said she had not missed a discrepancy in nine years. Nobody could know that, not even Madhavi. A discrepancy she missed was one she never saw.
In April 2025 a review system began reading her area’s records. It read every batch record, every report of a deviation from procedure and every monitoring log, in hours rather than weeks, and it flagged what did not fit. By July the daily review that had filled her analysts’ weeks had shrunk to a fraction of itself.
For the first two months Madhavi checked again every record the system had cleared. Her checks matched. They matched every time. This call goes to the machine, and it should be said plainly. On the work it was given, the system was at least as careful as she was, much faster, and never tired in the afternoon.
She kept checking anyway. Part of the reason was grief. A job bundles four goods: income, practice, standing and voice. Madhavi kept her income. She was losing her practice, the daily reading that had made her eye. She was losing her standing as the person whose desk the hard cases came to. But she also believed something definite. She believed there was a part of the work, perhaps a tenth, that the system could not see. It might be a pattern across many records that no single record breaks, or an investigation that found what it expected to find.
She might have been right. Two psychologists, Daniel Kahneman and Gary Klein, gave a test for a belief like hers: an expert’s sense can be trusted where the world is regular and the feedback is quick and honest.1 Her world was regular. Its feedback was poor. When she passed a batch, she seldom learned what became of it. Years of experience give an eye only where they come with feedback. Elsewhere they can give confidence instead.
So in the autumn Meera tested the tenth. She had asked experienced people to judge the machines once before, early in 2025; this time she followed one person’s claim for a quarter. From November 2025 to January 2026, every flag Madhavi raised that the system had not raised was written down and followed up. Madhavi said what she could not prove: “I cannot tell you why. I want it looked at.” Meera decided that it would be. Most of the flags came to nothing. Two did not. One was a slow rise in the particle counts of a clean room, inside the limits, which crossed them three weeks later. The other was a deviation report whose investigation had stopped one step too early. Neither broke a limit, and the system had applied its rules correctly. The two cases showed that the rules did not hold everything the work needed noticed.
The log is not a verdict on Madhavi. It says that her eye sometimes saw what the rules did not, and less often than she believed. It also did something more useful. Part of what passes for depth is a record nobody kept, and a kept record can become a rung (Chapter 10). Before November her tenth lived only in her. Once it was written down, the two real cases could be taught. They could be taught to the system too, and its rules widened to catch the next slow rise in a clean room’s particle counts. That would be a gain, and the case for Madhavi’s seat would lose nothing by it. The seat rests on her signature, and on a place where someone still learns to see what the rules do not yet hold.
Notice what the quarter did not change. Madhavi still signed. At NovaChem a batch left her area only with a quality signature, and hers was the one. She still had the right to stop a batch, and she would still have to explain it if a bad one got out. Her name and her power had stayed. What was going was the practice that had made her able to use them: thousands of records a year, read by hand, that had trained her eye. In five years she would still be signing, on a thinner practice each year. A person can remain accountable long after the organisation has removed the practice that made her able to carry it. Nobody plans this. It happens when a task moves to a machine and the signature stays where it was. Following Madhavi’s flags for a quarter kept some of that practice alive, though nobody had said that was what it was for.
Then there were her analysts. Two of them had joined in 2024. The daily review was where they had been learning to read a batch record, and now the machine did it. This is the missing rung: the work a beginner learns on goes to the machine, and ten years later nobody has learned how to check the machine. The rung was missing here, in Meera’s own division. Basel’s review did not ask about it. In the form the analysts were six rows marked change, with no word about what they would learn instead.
The oldest advice on this that the book has found is Tamil. The Tirukkuṟaḷ, the classical book of Tamil maxims, has a chapter that V. V. S. Aiyar’s translation titles “Testing and Employment of Men”. It is about fitting people to work. One verse says to judge first what a person can do and what work fits them. The next adds a duty that most job descriptions leave out: “After thou hast decided that a man is fit for an office, make him worthy to fill it.”2 The verses are counsel to a king about his servants, and I borrow only their order, not their court. But the order is worth having. First comes the fit between this person and this work. Then comes a duty for whoever chose: to make the person worthy of the office, which in a workplace means giving them the means to fill it.
A job description written for another century lists tasks. Madhavi’s, written as tasks, had shrunk. Written as positions, it had not. Written as a rung, it was empty. That was what needed redesigning, and none of Basel’s four choices named it. The work that still needed a person had no role attached to it, and attaching one is the employer’s work. Roles built around tasks a machine now does should be rebuilt around who answers and where the next person learns. Whoever rebuilds them owes the people in them the means to fill them, and that includes the practice that keeps a signature honest.
What would prove this wrong?
I claim that a signature can outlast the practice that made it honest. A person who mostly confirms what a machine has passed will, over years, see less on her own. That is checkable. If people who signed machine-checked work for years kept their own eye as sharp as colleagues who still did the work by hand, I would be wrong. Madhavi’s thinning practice would then cost her nothing that matters.
The best evidence so far points my way, and it is thin. A study told earlier in this book looked at doctors in four endoscopy centres in Poland who had begun using an AI tool. When they worked without it, they found precancerous growths in about 22 of every 100 patients, against about 28 before the tool arrived.1 But the study looked back at fewer than 1,500 patients over about six months, and it could not rule out other causes. Its authors wrote only that such exposure might lower the rate.
What would change my mind: a trial that tests reviewers without the machine after a year of confirming its findings, against colleagues who kept doing the work themselves. Batch review in a drug company would be a fair place to run it.
19.2 The Decline of Traditional Middle Management
“The Antwerp situation,” said Wouter Brinkman, and four people around a glass table in Rotterdam moved in their chairs.
It was the weekly operations review of a logistics company, on a Monday morning in late September 2025. It lasted ninety minutes, as it had every week for six years. Wouter managed four team leads. Katrien went first, as she always did, with her numbers memorised so that she could look at him while she gave them. Shipments handled. Problems solved. The delays at the port of Antwerp that had broken two delivery promises to clients the week before. The quietest of the leads went next, reading from his tablet. Then came the other two, one of them covering for Dirk, who was on leave. Wouter asked the questions he had asked for six years. Why had the problems risen? Was Antwerp solved, or only contained? Who was covering for whom? They were good questions. On Thursday he would turn the replies into a report for his own director.
His director had not needed that report since June. In June 2025 the company had given the director a dashboard. It gathered every number Wouter’s leads reported, wrote its own summary each morning, flagged the problems and drafted the follow-up. By Thursday the director had known everything in Wouter’s report for days. This call goes to the machine too. On the numbers, the dashboard was faster, fuller and earlier than the meeting.
Nobody had told Wouter.
Chapter 6 described the layer Wouter belongs to, the people between those who decide and those who do the work, and said that machines compress it. It ended with a promise: “The people who remain in it will be those whose value was never mainly in passing information along, but in being the one others can ask why.” Wouter is such a person. That does not decide whether he remains. Someone else decides that, and for three months that someone had said nothing.
Look at what had happened to his position. Chapter 7 said that a firm can empty a position “by leaving the name and taking the power”. Wouter still had the meeting, the report and the title. What the report had been for, putting the picture in front of the people who decide, had gone elsewhere in June. For three months he held an empty position and did not know it. He had no hearing and no say, and for three months the only notice he had was silence. Silence is a design decision too. A handoff with nobody’s name on it fails, and so does a change that nobody announces. That half of the story is the company’s. What it owed Wouter was an honest account of how the work had changed, given in time for him to say what the work still needed, and not his old job unchanged.
Wouter knew things the dashboard did not. Katrien was thinking about a move to Berlin, where her partner had been offered a job; she had told nobody but him. The quiet lead’s team had been carrying a colleague through a hard year, covering shifts without reporting it. And a new routing system on the warehouse floor, which the dashboard counted as a success, was breaking old agreements. Over years the teams had settled among themselves who worked the heavy aisles.
It is tempting to say that he knew these things because he was there. The point is simpler and more serious. As of 2026, a system that read a company’s calendars, rosters and messages could guess some of what Wouter knew. Whether it should is another question. Katrien told a person, not a company. Some of what a manager knows is held in trust, and a role is partly a place where such trust is kept. Being able to infer a confidence does not make it the company’s to collect. Putting Katrien’s plans on a dashboard would not complete the picture. It would break the trust that made the picture possible.
The routing trouble was different, and it shows what Wouter’s role still owed. The trouble was real, it was costing goodwill on the floor, and somebody had to take it to the people who had chosen the system. In this chapter’s terms, that is a position: someone who answers for what a change does to the people who live with it. The Monday meeting had never been the place for it.
The print edition was hard on Wouter. It said that he “hides behind the meeting”, that the meeting was a “fortress”, and that whether he could lead was “a question of courage”. Weigh that against the ordinary explanation. He was measured on the meeting, and nobody had told him that it was no longer needed. Some of what looks like hiding is a person doing well the job he was given, after the job has quietly changed.
In November 2025 his director told him that the weekly review would end in the new year. By the autumn of 2026 Wouter still managed the four leads. The review had become a monthly hour about people, not numbers, and he had taken the warehouse agreements to the people who ran the routing system. His company had not said whether that hour and that errand add up to a role, or only to a courtesy.
19.3 The Middle Manager’s Pivot: From Controller to Cultivator
Meera’s twelve managers were four in quality assurance and quality control, three in supply chain and logistics, three in manufacturing and two in procurement and facilities. Most had been promoted because they were excellent at the work their teams did. They could review their teams’ work because they understood it better than anyone in the team. The machines had taken much of that reviewing. Meera could see what it did to them: a defensiveness about their teams’ output, and extra hours spent checking what the system had already checked.
In October 2025, over two weeks, she sat with each of them and asked one question: “What does your team need from a human leader that an AI cannot provide?”
The question had a flaw she did not see. It asked for what machines cannot do, and that list gets shorter every year. A question built on it must be asked again each year, with less left over each time. A better question would have asked what a team needs someone to answer for. Her managers took up the better question anyway. Their teams needed someone who saw the person behind the number. They needed someone who noticed when a colleague had stopped caring, before the output showed it. They needed someone to handle the friction between departments. And they needed someone to make the call when the machine’s recommendation and the team’s experience disagreed, and to stand behind it. None of this was in any job description. None of it had been counted in any review. Every manager was already doing it, in the time left over from the work the machines had now taken.
Meera called the change “from controller to cultivator”. A controller manages the flow of work, checks its quality and watches the timelines. A cultivator, she said, grows people. From November 2025 she rebuilt the twelve roles around that idea. The weekly status meetings went, and conversations about each person’s development took their place. Each manager would be judged by the growth of their people, measured by how complex the decisions were that team members could make on their own.
The image is older than Meera’s phrase, and it carries a warning. The Chinese philosopher Mencius, in the fourth century BCE, told a story about growth. In James Legge’s translation: “Let not the mind forget its work, but let there be no assisting the growth of that nature. Let us not be like the man of Sung. There was a man of Sung, who was grieved that his growing corn was not longer, and so he pulled it up. Having done this, he returned home, looking very stupid, and said to his people, ‘I am tired to-day. I have been helping the corn to grow long.’ His son ran to look at it, and found the corn all withered.”3
Mencius names two failures. Some people neglect growth: they “let it alone”, and “they do not weed their corn.” Others pull. “What they do is not only of no benefit to the nature, but it also injures it.” He is speaking about a person’s own moral growth, not about managers, and I borrow only his two failures. But the borrowing shows something the print edition missed. A cultivator who decides what the plant must become is a controller with a gardener’s tools.
That is what Meera had built for her managers. For each of them she kept a monthly check-in, a set of milestones and a progress tracker. Usha Iyer, the manager her pencil had marked in September, was measured against it like everyone else. Usha had spent twenty years being measured against specifications, and she recognised one.
At the end of January 2026, in her third monthly check-in, Usha said it. Her sentence is about one rubric, Meera’s, and not about every measure an organisation keeps. Inside that boundary it is exact:
“You are asking me to stop controlling my team. But you have not stopped controlling me.”
Twenty years and a strong record made it safe for Usha to say. A manager in her first year might reasonably have kept quiet, and the silence would have said nothing about her character.
Meera was measuring Usha’s growth against Meera’s idea of growth. She was pulling the corn.
The Kuṟaḷ’s chapter on employment has an order for this, and Meera had not followed it. “Determine first the capacity of the servant and the work for which he is fit: and then leave him in responsible charge of the same.” Leave him in charge. The same chapter then ends: “Let the prince oversee everything every day: for there will be nothing wrong with the country so long as there is nothing wrong with the officers of the State.”2 The Kuṟaḷ does not pretend that the king stops watching. He watches every day. The medieval commentator Parimēlaḻakar adds that he does it within himself, without destroying the officer’s urimai, his standing in the office. And he watches conduct, whether his officers are honest, not whether they are growing into his idea of them.
The two lamps do not quite answer the same question. The Kuṟaḷ is about employing people already formed: choose them for the fit, give them charge, and watch what they do. Mencius is about forming: growth keeps its own time, and the helper who pulls kills what he tends. Read with Parimēlaḻakar, the two come close, because his king watches without taking back the charge he gave. What tension is left lies in one place. The Kuṟaḷ still has the prince look every day, while Mencius warns that even a helping hand can harm what is still growing. Meera’s tracker was neither the prince’s quiet look nor the farmer’s weeding. It reached into the growing itself, and that is what Usha named. Here Mencius goes deeper than Meera’s word. His farmer is tending his own corn. In his story, growth belongs to the one growing.
Usha’s sentence did not call for a new idea. It called for a different arrangement. The philosopher Onora O’Neill called it intelligent accountability. It checks what a person is accountable for, without destroying the trust and the good work that the checking is meant to protect.4 It still checks. The difference is not whether one measures. It is what a measure is for, who chose it, and whether the person measured can dispute what it is taken to show. Meera’s tracker failed on all three. It measured growth, which was not Usha’s to deliver on a schedule; Meera alone had chosen it; and it gave Usha no way to say that a reading was wrong. Goodhart’s law, named after the British economist Charles Goodhart, names a further danger: when a measure becomes a target, it stops being a good measure.5 A count of “complex decisions made alone” invites managers to hand hard decisions down too early, or to call easy ones hard.
A week after the check-in, Meera gave Usha different terms, and said them out loud. Basel required one line a quarter on each manager. She showed Usha the line, and a place beside it where Usha could write that it was wrong; whatever Usha wrote there would go to Basel with the line. Meera would still look at what Usha was accountable for: deviations closed late, batches held, complaints. What Usha’s team learned, and how, was Usha’s to decide. That was the Kuṟaḷ’s order: leave her in charge, and watch her conduct without taking the charge back. Parimēlaḻakar’s king keeps his watching to himself; Meera said hers out loud, a choice the old text does not make for her. Asking inside a frame you control is still control, and a frame that someone above you controls stays theirs.
What Usha did with the room she was given was not what anyone had planned. From February 2026, each of her analysts took one deviation report a week that the machine had produced. Before reading the machine’s explanation, they wrote their own. Why would this be flagged? What data drove it? Where could the conclusion be wrong? Then they compared their reasons with the machine’s, and with one another’s. The machine was almost always right. The exercise was for the analysts, and Usha did not judge it by whether they beat the machine. They seldom did. She asked other things. Could an analyst explain why a report had been flagged? Could she find a flaw in the machine’s reasoning when there was one? Could she work a new case without the machine’s framing in front of her?
Earlier chapters had people check, or ask to check, a sample of what a machine had passed. In Pune a manufacturing director kept a small team checking a rotating sample of parts by other means. In Tiruppur a floor supervisor at a garment factory asked that two inspectors moved to packing do the same; the owner had not decided. Both were meant to catch what the machine missed. Usha’s practice had another purpose: formation. It made the machine a training ground, and it asked for a first attempt before the machine’s answer. It rebuilt the rung the machine had taken. Some work has to be kept because the work is how people learn, even when a machine does it faster.
It cost something, and someone paid. Two hours a week for each analyst came out of the quality team’s time. For two months the team closed deviations more slowly, and the monthly report to Basel showed it. Meera approved the hours and wrote down why. The hours a machine saves belong to whoever owns them. Here, for once, the owner spent some of them on the next generation.
In March Madhavi’s team began the same practice. Her two real flags from the winter became two of its first cases.
By the end of March 2026 the twelve had settled, though not in the way the September form had sorted them. Three took to the new work at once; most of it they had always done in the margins. Five came to it over months. Two struggled. Madhavi was one of them. She found part of her footing in Usha’s practice, reading the machine’s reasoning with her analysts as she had once read a batch record. Two did not want the new work, or could not do it in the time NovaChem allowed. From April 2026 Meera moved them to technical posts elsewhere in the company, where knowledge of the process was the job. She told each of them herself. Neither of the two was the manager her pencil had marked. The one it had marked was the one who had told her what her rubric was.
At the end of March Meera filled in the final return for Basel. The four boxes had not changed. Of the twelve managers’ rows, ten said change, and two said end, for the two whose holders would move in April. No box had room for a rung, or for a trust. She could loosen her own rubric. She could not change the four boxes.
The two lamps judge her decision about the two managers differently. The Kuṟaḷ would call it sound: fit the work to the person, and employ each where they fit. Mencius would ask whose clock had judged them. The print edition said they “could not develop within the timeframe the organisation needed” and called the decision “right and painful”. “The timeframe the organisation needed” is the man of Sung’s complaint about his corn. Both readings may be true at once. The decision was Meera’s to make, and she made it. It was painful. How things turned out cannot settle whether it was right, and this edition does not try to settle it.
Nor was the new arrangement finished. In January 2026 it had already failed in one place, in a way Chapter 30 tells. The print edition ended this section with a bet on the cultivator model, and said that “this book thinks she is right to make it”. This edition makes no bet. Meera learned one thing in this chapter, and Usha told it to her: asking people what they need, inside a frame you still control, is still control. The cultivator’s part is the weeding: time, honest feedback, real cases, and protection for a good refusal. The growing belongs to the one growing.
19.4 Creative Directors and Operators
To set creative directors against operators turns two kinds of work into two kinds of people on opposite sides. Chapter 9 refused that: the architect’s question, what this work is for, “is open to anyone, on any day”; it is a way of working, not a rank.
The studio in this story is a composite, drawn from several firms in Jakarta and Kuala Lumpur. It made campaign material for brands: films, banners, posts, and the many versions of each that every platform needs. At the start of 2025 it had about sixty people, including writers, designers, video editors, social media managers, coordinators and account staff. Over that year, as machines took over most of the making of versions, the studio rebuilt itself around about twenty. The machines made the making cheaper. Whether that required sixty to become twenty, or the owners chose it as a use of the savings, was the owners’ decision.
Adi had worked there for five years in production design. He was the person you called when a client needed thirty versions of a banner by the end of the week, each sized for a different screen and each exact. His speed was famous in the office. By the middle of 2025 a machine made a hundred versions while Adi was opening his files, and most of them were good. This call goes to the machine as well.
Adi did not become a creative director. He did not want to. In about October 2025 he took the severance pay the studio offered for leaving, and found work at a printing company, where hands on machines still made the thing. Run his story through the four goods. His income dropped. His practice moved to another trade. His standing among designers went, and so did his voice in the studio, because he was no longer in it. The studio lost his eye for detail and his knowledge of its clients. It will not get them back.
The twenty who stayed were organised around clients. Each client had a creative director, who decided what the work was for; a relationship manager; and a production specialist, who ran the machines. The directors were not, in most cases, the old heads of department. They came from every level. They had three things in common. They could see what a client needed before the client could say it. They could tell when the machine’s output was good and when it only looked good. And they had the nerve to propose what nobody had asked for.
A building needs both the architect and the bricklayer, and the hierarchy between them is a social convention, not a natural law. But the print edition promised too much. It said that “the market pays for direction”, and that people who build the skills of direction are “positioning themselves wisely”. The studio’s new shape drew the old line between deciding and making again: twenty seats where purpose was decided, and no seats below them. The market may pay for direction, in this studio, this year. A book cannot promise it. There were twenty seats, and wisdom did not make a twenty-first.
Now look at where the twenty learned their eye. Every one of them had learned to tell good work from work that only looked good by doing the work. That meant years of banners, edits and versions, much of it in Adi’s kind of job. The new studio had no such job. It hired no juniors. It kept the people the old structure had formed and removed the structure that formed them. The shape is old. Each firm hopes that another firm will train the seniors it needs, and when every firm hopes the same, nobody trains them. The missing rung is a young person’s problem, and it is also how a studio of twenty eats the future it depends on.
The machine can also rebuild the rung. In a study of customer-support agents, an AI assistant raised the number of problems they solved per hour by 15 per cent on average, and the least experienced gained most.6 A machine can pass the habits of the best workers to the newest, but only where there are new workers. And a machine beside a junior does not rebuild the rung by itself. The junior needs real work, room to fail, and someone who corrects her. Whether this studio hires two juniors and gives them production work with the machine beside them is a decision. The owners will make it, and it will be theirs either way.
There is an honest objection to the studio’s new shape, and it should stand. Making things is not as cheap as the studio’s plan assumed. As of 2026, the last part of quality, the detail that separates professional work from adequate work, still needs a trained eye in many trades. The studio works only while its directors can see the gap between good and nearly good. If nobody below them ever does production work, that eye is no longer being made. The structure does not guarantee the eye. It uses up the eye it inherited.
19.5 Intelligence Governance: Who Answers for the Machines?
It is natural to ask who watches the machines. In the case at the centre of this section, a machine did the watching. What was left was the question of who answers.
Reports of harm from patients in NovaChem’s clinical trials around the world came to a case-processing team in Hyderabad, run within Meera’s operations for the company’s drug-safety function in Basel. This work is called pharmacovigilance, or drug safety: collecting and judging reports of side effects, so that a danger in a new medicine is seen early. A model sorted the incoming reports and gave each a score: how likely it was to signal a real safety problem. The model was good. It found links between dose, patient group and harm that people reading the reports one at a time had missed.
In July 2025 an automated audit, which checked the model every quarter, flagged something. The same event scored differently depending on where the trial site was. Reports from sites in rich countries received higher scores than the same events reported from sites in Nigeria, Kenya, India and the Philippines. In the World Bank’s grouping, all four were lower-middle-income countries in 2025.7 Nobody had programmed that. The model had learned it from history. Reports from those sites had often been less complete and more often challenged in review, and the model had learned to trust them less.
This call goes to the machine, and it is the most important one in the chapter. The audit caught what nobody at NovaChem had seen, though people worked beside the model every day.
The model was right about the records. It was wrong about the patients. A safety signal from a patient at a clinic in Kenya matters as much as the same signal from a hospital in Switzerland. A model that learns to discount it has learned the habits of its history, not the duty of drug safety.
The audit reported a pattern. It did not say what the pattern meant. Meera read it and asked a different kind of question: “Are we treating all patients equally?”
The print edition built its argument on that difference. “An AI can optimise for the metrics it is given,” it said. “It cannot ask whether the metrics are the right ones. This is not a limitation that better engineering will solve.” That is withdrawn. As of 2026, a model asked whether a weighting like this treats patients fairly can raise the same doubt Meera raised, and set out the arguments on each side. Asking is a task, and tasks carry dates. What carries no date is the Prologue’s third never: “Being intelligent will never give a machine the right to decide for people.” It adds that knowing best is not the same as having the right to decide. The question was never whether something could ask. It was who had the right to decide what NovaChem owed its patients, and who would answer for the decision.
That right was not Meera’s. At NovaChem the threshold at which a safety signal is investigated belonged to the drug-safety physicians, and they sat in Basel. Meera’s part was to ask the question, put it on the record, and take it to the people who held the seat. In August 2025 the data scientists spent three weeks building a rule. Under it, the same event would get the same score wherever it was reported, and signals from every site would be judged at the same threshold. Poor records would be dealt with separately, by helping the sites to report better, not by trusting their patients less. The physicians accepted the rule in writing, with Meera’s question attached. The cost was known in advance: more false alarms from sites with patchy records, and more reports to investigate. A fairer rule creates work, and the work has to be paid for, not quietly moved onto whoever is nearest. Meera put a written request to Basel for two more case reviewers, and got one. The rest came out of her teams’ hours, and she wrote down that it had.
The positions, then, sat with different people in a chain. The machine flagged, and that was a task. Meera asked, and answered for what the rule cost her teams. The physicians decided. The patients would live with the result, and they had no say in it, which is why every other position had to be held.
The trade-off was real, and it is no failure of engineering. In 2016 three researchers, Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, took three common conditions of fairness for a risk score. They proved that “except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously”.8 The proof concerns particular statistical conditions, not every meaning of fair. NovaChem still had to decide what equal concern for patients required in its own safety work. Choosing which conditions to keep is a choice of values, made by people who can be asked why.
After the flag, Meera proposed a team that would keep asking. Quality wanted to own it, and so did compliance and the technology department, and each would have made it about its own work. She argued for a team that stood apart: reporting to the site’s leadership, and accountable to the company’s global ethics board. It was approved in November 2025 and began work in December. It had three people. One was a former quality analyst who knew the processes. One was a data scientist who had moved from building models to testing them: Revathi, whose work Chapter 31 follows. The third was a bioethicist with ten years in academic medical ethics, who brought a way of reasoning when values conflict. Among the powers written into its terms was this one. It could require the owner of any system to reply in writing to a finding, and to test the system again. A team that can only record concerns becomes the place where an organisation parks its unease.
The print edition called them “The conscience of the system.” A conscience cannot be a department. Three people can keep a question asked. They cannot ask it on behalf of four hundred, and a team named as the conscience can let everyone else stop asking.
The Kuṟaḷ’s verse about the prince who oversees everything every day, quoted in 19.3, fits this team better than the word conscience. The verse is about a king and his officers, and applying it to software is my reading, not the text’s. But notice whom the prince watches. He watches the officers, because the country goes wrong when they do. The adverse-event model went wrong through decisions people had made about its data, its scores and its use. Watching machines is, in the end, watching the people who set them up and use them. So the team’s work is to make sure that, for each machine, someone in the right seat answers. And the team must explain itself too, to the site’s leadership and the ethics board. It will not catch everything, or catch everything in time.
The objection to all this is that governance slows things down. It does, and sometimes it should. Chapter 17 asked who chose the purpose put into a machine. In drug safety the purpose is finding harm, and speed serves it. A drug company that trusts some patients’ reports less in order to move faster saves no time of its own. It spends other people’s safety.
Ending: The Second Year’s Form
A year after the form, in September 2026, Meera sat in a meeting room on the seventh floor with the governance team and two of her managers. They were reviewing another model. This one helped NovaChem’s trial-support team in Hyderabad choose clinics to approach for patients in a new clinical trial. It was a model for recruiting patients, not staff.
The team had found a pattern that might be nothing. The clinics the model suggested drew patients who were, on average, younger and more urban than the people who had the disease.
One of the managers asked the question. She was the supply chain manager Chapter 16 met, with two decades at NovaChem, and in April she had moved into quality operations. “What does ‘fair’ mean for recruiting patients?” she said. “The same number from every group? A number that matches who has the disease? Or something else?”
Nobody in the room had the right to answer that alone. The team wrote the question down and attached what it had found. It sent both to the trial’s steering committee, whose decision it was, and to the doctors who would have to explain the trial to patients. Sending it on was a step, not the end. Meera did not say what she thought.
That week Basel sent the form for the second year’s review. The last column had the same four choices.
Practice
What each task carried. Take a role you hold, or one you manage. List the tasks in it that a machine now does, or soon could. Beside each, write what that task carried besides the work. It may have carried an answer: someone who had to explain the result. It may have carried a decision: someone with the right to say yes or no. It may have carried a rung: someone who learned the work by doing it. Or it may have carried a trust: something people told a person that they would not tell a system. Mark each one that has no holder now, and write who could give it one, and whether they know. If you manage people, ask one of them two questions. Ask Meera’s question, made better: what does your team need someone to answer for? Then ask the reverse of Usha’s sentence: where am I still controlling you? Write the replies in their words. Choose one thing you will stop checking, tell them, and say what you will still check and why. If you manage nobody, do not assume it is safe to ask your manager the same; keep the page, and show it only where that is safe. This takes an hour, and if the hour is your employer’s, note who gave it. None of this will keep a role for you. It shows what the role was carrying.
Words to keep
- bharatanāṭyam: a classical dance of South India.
- controller and cultivator: the print edition’s two kinds of manager, one who controls tasks and one who “grows people”; in this edition a cultivator tends the conditions of growth and does not pull.
- empty position: a position kept on paper after the power or the ability to hold it has gone (Chapter 7).
- fairness trade-off: the proven conflict between common conditions of fairness for a score; choosing among them is a choice of values.
- missing rung: the loss of the work beginners learned on, and with it the people who could later check the machine (Chapter 4).
- pharmacovigilance: drug safety; collecting and judging reports of harm from medicines, so that dangers are found early.
- role: in this book, three things: its tasks; its positions (who answers, who decides, whose life it touches); and its rung, where the next person learns.
- urimai: in Tamil, a right or rightful standing; in Parimēlaḻakar’s comment on Kuṟaḷ 520, the officer’s standing in his office, which the king’s daily watching must not destroy.
Sources
- Daniel Kahneman and Gary Klein, “Conditions for Intuitive Expertise: A Failure to Disagree,” American Psychologist 64, no. 6 (2009): 515–526; see Chapter 3. For “What would prove this wrong?” in 19.1: Krzysztof Budzyń et al., “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study,” Lancet Gastroenterology & Hepatology 10, no. 10 (2025): 896–903: a retrospective, observational study at four centres in Poland; 1,443 patients had standard colonoscopy between 8 September 2021 and 9 March 2022, 795 before AI was introduced and 648 after; the adenoma detection rate fell from 28.4 per cent (226 of 795) to 22.4 per cent (145 of 648); the authors conclude that continuous exposure to AI “might” reduce it (abstract, via Europe PMC, checked 4 November 2026); the study is told in Chapter 3.
- Tirukkuṟaḷ 517, 518 and 520 (chapter 52), trans. V. V. S. Aiyar, The Kural or the Maxims of Tiruvalluvar (Madras, 1916; Standard Ebooks), chapter LII, “Testing and Employment of Men”. Chapter 7 quotes 509 and 510 from chapter LI, “Testing of Men for Confidence”. The paraphrase of 517’s first clause in 19.1 is mine. Parimēlaḻakar’s commentary on 520 ends: avaṉ urimai aḻiyāmal taṉṉuḷḷē ārāyntu pōtuka, “let him go on examining within himself, without destroying his standing” (my rendering; text at thirukkural.io).
- Mencius 2A:2 (Book II, Part I, chapter 2, paragraph 16), trans. James Legge, The Works of Mencius, The Chinese Classics, vol. II (Internet Sacred Text Archive). The passage concerns nourishing one’s own “passion-nature”; the application to managers is mine.
- Onora O’Neill, A Question of Trust: The BBC Reith Lectures 2002 (Cambridge: Cambridge University Press, 2002), Lecture 3, “Called to Account”; see Chapter 12.
- Marilyn Strathern, “‘Improving ratings’: audit in the British University system,” European Review 5, no. 3 (1997): 305–321, at 308; see Chapter 9.
- Erik Brynjolfsson, Danielle Li and Lindsey Raymond, “Generative AI at Work,” Quarterly Journal of Economics 140, no. 2 (2025): 889–942; the figure is the published version’s; see Chapter 13.
- World Bank, “World Bank Country and Lending Groups”: in the classification for fiscal year 2026 (from July 2025) Nigeria, Kenya, India and the Philippines are lower-middle-income economies; in the classification for fiscal year 2027 (from 1 July 2026) the Philippines is upper-middle income (datahelpdesk.worldbank.org).
- Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, “Inherent Trade-Offs in the Fair Determination of Risk Scores,” in 8th Innovations in Theoretical Computer Science Conference (ITCS 2017), LIPIcs 67 (2017), 43:1–43:23; first posted as arXiv:1609.05807 (19 September 2016).
Named people in this chapter are composites, and their conversations and circumstances are constructed; the studies, laws and figures cited are real and sourced.
Revised from the print edition (2026)
- What the print edition claimed: that when the tasks of a role go, what remains is “what was always underneath: the judgement, the relationships, the capacity to make meaning from ambiguity”, made valuable by consciousness; that a quality manager “had not missed a discrepancy in nine years” and saw a pattern “too subtle for algorithmic detection”, in a chapter whose own re-checks “always matched”; that she would become “one of the most valuable people in the division”; that a manager “hides behind the meeting”, that the meeting was a “fortress”, and that whether he could lead was “a question of courage”; that he knew what he knew “because he is present”; that a team auditing the machine learned “the recursive awareness that this book calls consciousness” by examining “an intelligence that could not examine itself”; that after Sunitha’s sentence “the model was sound” and only the method changed; that two managers “could not develop within the timeframe the organisation needed”, a decision “right and painful”, after which “Twelve became ten. The arithmetic was clean.”; that “this book thinks she is right to make it”; that “the market pays for direction” and those who learn it are “positioning themselves wisely”; that “The structure does not guarantee consciousness. It only rewards it.”; that “An AI can optimise for the metrics it is given. It cannot ask whether the metrics are the right ones. This is not a limitation that better engineering will solve.”; that “The model was not wrong”, in a case of reports from “low-income countries”; that Meera accepted the safety trade-off herself; and that three people were “The conscience of the system.” It also named Sunitha Rao, Deepa and Jan, gave the Rotterdam manager the name of a real logistics scholar in that city, drew Meera’s role map on a third four-colour whiteboard and had her erase it, gave her the same lesson twice, broke its own calendar (a “late November” coda six months after an April move), and placed its studio partly in Manila.
- What I now hold: a role is three things: tasks, positions (who answers, who decides, whose life it touches) and a rung where the next person learns. A machine takes tasks; positions stay and are held or emptied by people with authority; the rung goes unless someone rebuilds it. A job does not become anything by itself. Madhavi’s re-checks matched; her “tenth” was tested for a quarter and was real twice, less often than she believed; written down, it could be taught. Her analysts were the missing rung inside the division. The Kuṟaḷ adds the employer’s duty: having judged a person fit for an office, “make him worthy to fill it”. Wouter held an empty position for three months because nobody told him; some of what a manager knows is held in trust and should not be on a dashboard; Chapter 6’s promise is answered: being the one others can ask why does not decide who remains. A cultivator who decides what the plant must become is a controller (Mencius’s man of Sung); the Kuṟaḷ leaves the formed in charge and watches their conduct every day, without (on Parimēlaḻakar’s reading) taking back their standing; the two answer different questions and come close, the remaining tension being daily oversight of a person still growing. Asking inside a frame you control is still control. Usha’s practice rebuilt the rung and cost hours that someone paid. The studio of twenty used up the eye its old structure formed. As of 2026 a machine can ask whether its metrics are right; it has no right to decide what a company owes its patients (the third never). The model was right about the records and wrong about the patients; the threshold belonged to the drug-safety physicians; fairness conditions conflict (Kleinberg, Mullainathan and Raghavan), so choosing among them is a choice of values. A conscience cannot be a department; watching machines is watching the people who set them up. The three relations of a position can sit with different people (a quality manager signs for a batch; physicians decide a safety threshold; patients live with both). A person can remain accountable long after the organisation has removed the practice that made her able to carry it: Madhavi keeps the signature and the right to stop a batch on a thinning practice. The system applied its rules correctly; the rules did not hold everything the work needed noticed. Silence is a design decision; a company owes an honest account of the changed work, not an unchanged job. What matters in measuring is what a measure is for, who chose it, and whether the person measured can dispute its reading. Some work must be kept because the work is how people learn; a training exercise is judged by explaining, finding flaws and working a new case without the machine’s framing, not by beating the machine. A machine beside a junior does not rebuild the rung by itself, and sixty to twenty was a choice about the savings. The fairness proof concerns particular statistical conditions, not every meaning of fair. A fairer rule creates work that must be paid for. A governance team needs a power (a written reply and a re-test), or it becomes the place where unease is parked. Sending a question to those who must decide is a step, not the end. A machine that decides with nobody signing leaves a position empty; it does not hold it. Madhavi’s two cases can be taught to the system as well, and her seat rests on the signature and the rung.
- What changed my mind: the Prologue’s rule against defining people by what machines cannot yet do; Chapter 4’s missing rung and four goods; Chapter 7’s positions and the empty position; Chapter 9’s refusal to make the architect a rank; Chapter 10’s finding that depth fills a position and does not grant one, and that a kept record can be a rung; Chapter 11’s line between a decision and its outcome; Chapter 12’s question of whose hours; the Kuṟaḷ’s chapter on employment; Mencius’s farmer; the fairness proofs of 2016; the audit’s findings (the names, the calendar, the countries, the whiteboard, the repeated lesson); the reviews of draft 1.
- What survives: the role architecture review and its four boxes; “You are asking me to stop controlling my team. But you have not stopped controlling me.”; the cultivator, argued with; the batch record read like a score, and the photograph by the window; the two months of re-checks; the Antwerp situation and the dashboard nobody told Wouter about; Katrien’s secret; Adi and the printing company; “a building needs both the architect and the bricklayer”; the adverse-event bias caught by the automated audit, and “Are we treating all patients equally?”; the three-person team; the question of what “fair” means for recruiting patients.
- Section titles in the print edition: 19.4 was “Creative Directors vs. Operators”; 19.5 was “Intelligence Governance: Who Watches the Machines?”.