# AI-Proof: What Machines Can Never Be — Volume III: Transformation, Part VI: The Inner Revolution, Chapter 31: The Moral Responsibility of Builders

By Nithin Palal · 21 November 2026 · 47 min read
Section: Artificial Intelligence · Series: AI-Proof: What Machines Can Never Be
Source: https://nithinpalal.com/ai/ai-proof/chapter-31-the-moral-responsibility-of-builders

> A hiring system passed every fairness test, one trait at a time, and still marked down the people where three traits met. The manager who signed for it had done all she was asked. Good intentions settle a builder’s motive, not her debt.
*Part VI looks at the inner life. After fear in Chapter 29 and fragility in Chapter 30, this chapter turns to guilt: what a builder owes for harm she did not intend and did not see.*

← Previous: [Chapter 30: Anti-Fragility in an AI World](https://nithinpalal.com/ai/ai-proof/chapter-30-anti-fragility-in-an-ai-world) · Series hub · Next: [Chapter 32: The Examined Life in the Age of Machines](https://nithinpalal.com/ai/ai-proof/chapter-32-the-examined-life-in-the-age-of-machines) →

**A hiring system passed every fairness test, one trait at a time, and still marked down the people where three traits met. The manager who signed for it had done all she was asked. Good intentions settle a builder’s motive, not her debt.**

**In Brief: For readers who build, buy, approve or use systems that judge people, and wonder what they owe when one goes wrong. Chapter 31 follows the operations head of a drug company’s site in Hyderabad, who signed for a hiring system that passed every fairness test one trait at a time, and the data scientist who found the applicants it marked down where three ordinary traits meet. A Jain rule about walking carefully and the philosopher Bernard Williams’s lorry driver disagree about whether care is enough when a careful person harms someone. Together they help say what a builder owes the people her system judges: a search before it runs, a search after it has wronged them, and an oath sworn before someone who can hold her to it.**

**A builder’s good intentions settle her motive, not her debt: to the people her system judges and she will never see, she owes the care to look for them before it runs and, once it has wronged them, the work of finding them and putting right what still can be, a debt she shares with every hand that built it.**

## Opening: Eleven Pages on Her Desk

Meera had approved the system herself.

The analysis lay on her desk on the seventh floor, printed, eleven pages with a table on every page. It was late July 2026. Meera Reddy is the vice-president of operations at the Hyderabad site of NovaChem, a drug company with its head office in Basel. She had been with the company for fifteen years and vice-president for seven, and about four hundred people worked in her division. Clipped behind the first page was a copy of the form she had signed in June 2025, with her signature at the foot.

She had signed carefully. Two years earlier she had initialled a form for an earlier hiring screen without asking what it measured. In June 2024 that screen turned away a scientist whom the head of research had vouched for, and in July she removed it. Nobody went back to the hundreds of others it had turned away. So when the human resources department chose a replacement, she asked for more. The vendor showed her its fairness tests: by gender, by age, by education and by region, each tested on its own, and each clean. She asked for a document explaining every score in words a recruiter could use, because the old screen’s explanation had listed weights and told nobody anything they could use. The lawyers were satisfied. Then she signed.

The new system belonged to human resources. The company’s global head of human resources in Basel had bought it for every site. At Hyderabad it had scored applicants since September 2025. It estimated how likely each applicant was to stay eighteen months and earn satisfactory reviews, and below a certain score human resources sent a polite rejection without anyone reading further.

The analysis was Revathi’s, a data scientist on the site’s governance team. It said that every trait the vendor had tested was still clean. It also said that the system marked down one group of applicants far below what careful human readers gave them: people whose CVs showed three ordinary things at once.

So this chapter asks: **When a builder has done what the rules and the tests asked, and her system still harms people she never saw, what does she answer for?**

The print edition ended on knowing. Meera’s company had produced bias and knew it, and “in that knowing, in that uncomfortable, expensive, unfinishable knowing, lay everything this chapter means by moral responsibility.” This edition holds that knowing is where responsibility begins, not what it is. A builder answers for what she looked for and for what she failed to look for. Once a harm is found, she answers for going back to the people it reached, as far as they can be found. Many people built this system, and each answers for a different part. Sharing the debt among them does not make it nobody’s. Nothing in the argument depends on what anyone, human or machine, experiences. It turns on what people looked for, what they signed and whether they went back.

The title is kept, and its last word needs an argument. Meera did not build the system. A vendor wrote it. The human resources department chose what it should predict. Basel bought it. Meera signed for its use at her site, recruiters used its scores, and Revathi tested it. The European Union’s law on artificial intelligence already splits the builder in two: the *provider*, who makes a system, and the *deployer*, who uses it, with different duties for each. The philosopher Dennis Thompson’s name for the general difficulty, met earlier in this book, is “the problem of many hands”.¹ In this chapter a *builder* is anyone whose choice shaped what a system does to the people it judges. There were six kinds of builder in this story. The people it harmed met none of them.

The six held different powers. Basel bought the system for every site and, with the vendor, chose what it would predict. The vendor built it and tested it. The site’s head of human resources owned it at Hyderabad and decided how its scores were used. Meera signed for its use at the site. Recruiters acted on its scores. From December 2025 a governance team could test it and stop it at the site. Nobody held all of these powers. And nobody had been given the duty of going back to the people it judged, until someone took it.

Four questions are easy to run together, and this chapter keeps them apart. Did a person’s act help to cause the harm? Was she at fault? What does the law require of her? And who must now help to put it right? A builder can owe repair without being the one most at fault, and she can be at fault where no law asks anything of her. The European law’s *provider* and *deployer* are legal roles with listed duties. The *builder* of this chapter is a moral idea, and it is wider.

## 31.1 AI and Power: Who Benefits and Who Is Harmed

Revathi grew up in a village outside Tirunelveli, in Tamil Nadu. Her father drove an auto-rickshaw. Her mother taught at the government school. She studied at a state university and worked for five years at an IT services company in Chennai, where she took a master’s degree by distance learning in the evenings. She applied to NovaChem three times. The old screen turned her away in 2023 and again early in 2024. After it was removed, a recruiter read her third application, noticed a paper she had written on testing credit models for bias, and asked her in for an interview. She joined in early 2025, building models. Within a year she had moved to testing them, and in December 2025 she joined the governance team, three people whose work is to watch the site’s machines.

She was one of the hundreds nobody went back to. She came back by herself.

Power in this story sits in a few rooms. The measure the system would predict was chosen in Basel, by people who would never see an applicant from Tirunelveli. The applicants never saw the system. Most never knew it existed. This book has read Plato’s distinction between the user of a tool, who knows it best, and its maker, who knows less. It found no place in it for the person a tool is used on.² That person is the one this chapter is about. A job applicant is not the system’s user, and not its maker. She is its subject. She did not choose it, cannot see it and cannot argue with it.

The print edition put the difference between people and machines at a desk. A biased hiring manager, it said, affects dozens of candidates a year, inconsistently; a biased algorithm affects thousands “without the possibility that someone looks across a desk and sees something the data does not contain”. The algorithm, it said, “has no desk. It has no eyes.” That is withdrawn. The desk is where the old bias lived. In a study first published in 2007, researchers sent private employers in urban India applications that were equally qualified, with names that signalled caste or religion. The odds of a Dalit applicant being invited for interview were about two-thirds of a high-caste Hindu applicant’s, and a Muslim applicant’s about one-third.³ Another field experiment, in Delhi, found no caste gap for software jobs and a significant one for call-centre jobs, mainly against applicants from the Other Backward Classes. It found no gap against Muslims in either.⁴ The evidence differs by trade and by study. None of it supports a picture in which a person behind a desk is the cure for a biased machine.

What does change with a machine is scale and sameness. A biased manager is biased on his good days and his bad days, and in one office. A system applies the same score to every applicant, every hour, at every site that uses it. When it is wrong, it is wrong in the same way for everyone it reaches. Its makers gain the most from this sameness: one system, bought once, screening thousands. Its costs fall furthest from the room where it was chosen. They fall on people who were never in it.

## 31.2 Incentives and Harm: The Architecture of Unintended Consequences

The most important choice came first. It was made in one meeting in the first half of 2025, between the vendor’s project lead and NovaChem’s HR analytics manager in Basel. They agreed what the system would predict: whether a new hire would stay eighteen months with satisfactory reviews. Meera was not in the room. The governance team did not yet exist.

The measure sounded precise and reasonable, and it was both. It was also a theory of human value disguised as a technical specification. What matters is why that theory was chosen. The analytics manager’s own work was judged each quarter by early attrition: how many new hires left within their first year and a half. A system that predicted who would stay was a system that would make that number look better. Nobody intended harm, and nobody needed to. The incentive became the specification before anyone wrote a line of code.

This book has asked elsewhere who chooses what counts as success. Here the answer is plain: someone whose own success was measured by the same number. Such choices have a known history. In 2019 four researchers studied an algorithm widely used in American health care. It took the money patients had cost in the past as a stand-in for how sick they were. Because unequal access to care meant less had been spent on Black patients with the same needs, the system judged them healthier than they were.⁵ A stand-in carries the history of whatever it stands in for. In 2018 Reuters reported that Amazon had given up an experimental hiring tool that “penalized resumes that included the word ‘women’s’”.⁶ NovaChem’s old screen came five years after that report.

Consider what eighteen months of staying leaves out. It leaves out the applicant who does her best work in her third year. It leaves out the one whose CV has a gap because she cared for a parent, and the one who studied part-time while working. None of them appears in the measure, so the system cannot value them, by design.

Nor could the system have learned from its own outputs, as it is easy to imagine it did. A measure of eighteen months cannot report back on anyone hired eleven months ago. The real trouble is quieter. The outcome is seen only for the people who are hired. An applicant the system turned away is never hired, so no record ever shows that it was wrong about her. The error is unobserved, not unknowable. A careful reading by people, made without the score, is one way to look for it. This book has called this the problem of selective labels.⁷ Revathi is the old screen’s plainest example. It turned her away twice; she was hired only after it had gone, and nothing in its record ever showed that it had misjudged her.

The print edition also said that choosing among the mathematical definitions of fairness “cannot be computed. It must be judged. And judgement is an act of consciousness.” Part of that holds. In a paper published in 2017, three researchers proved that common definitions of a fair score cannot all be met at once, except in special cases.⁸ Someone must choose which to keep. The rest is withdrawn. Once a rule is chosen, the choice can be computed. The question is who had the right to choose the rule, and who will explain it to the people it judges.

What scale adds is repetition: the same wrong, for everyone the system touches, until somebody stops it. It is the difference between a mistake and an institution.

## 31.3 Designing for Human Flourishing, Not Just Human Productivity

This title needs one correction. “Flourishing” translates *eudaimonia*, Aristotle’s word for a whole human life lived well. A hiring screen does not give anyone such a life, and a company that set out to define one for its applicants would be overreaching. Aristotle himself denied slaves any share in it, a reminder that those who define flourishing have often left out the people they judged. What a hiring system can be asked is narrower. What does it do to the people it judges, who did not choose it?

Revathi’s audit was built to find out. Each month from February 2026, the governance team took a hundred applications at random. Two recruiters read each one with the name, photograph and score hidden, against a written description of the job. The recruiters were human resources staff, and their hours came from the governance team’s budget. Their readings were a yardstick, not the truth. A careful reading by two people can be wrong too. But it was a different kind of reading from the system’s, and where the two parted, someone should look.

For five months the two agreed on the whole, and where they differed the differences looked like noise. Then, in July, Revathi ran a search across eleven months of the system’s scores. A program compared rejection rates for every combination of the traits a CV could show, thousands of combinations, and listed the ones that stood out. No person could have done it by hand. Nobody had asked the vendor’s own tools to do it. One combination stood far above the rest: a first degree from a state university, a master’s degree taken by distance learning, and experience only in IT services. In the team’s monthly samples, the recruiters had rated these applicants about as highly as everyone else. The system put most of them below its line. Revathi checked it against July’s fresh sample before she believed it. Small groups give fragile results, and a search through thousands of combinations will almost always find a few by chance. Her search had used the system’s scores alone. The check used something else: July’s sample, read blind by the two recruiters after the search had named the group. It said the same. She wrote down how few people the group held in each sample, so that others could judge how fragile the result was. The team’s other two members ran the search again themselves before it went to Meera. Revathi traced part of the reason. In the records the system had learned from, a number of earlier hires with backgrounds like these had left NovaChem within a year. Most had gone back to better-paid work in IT. The system had learned their leaving as a rule about everyone who looked like them on paper.

The strongest defence of the score is that it may have been right about staying. If people with these three lines on their CVs did leave sooner, a system asked to predict staying was doing its job. A prediction about a group can be true of the group. But the score made the group’s past a verdict on each person in it, and none of them could argue. Most of those earlier hires had left for better pay, and pay was the company’s to set. Staying was also only the measure chosen in Basel. The recruiters, reading for the job, rated these applicants about as highly as everyone else. As I read it, the unfairness lay in a narrow measure, applied to each person as if she were her group’s past. The system was also predicting a fact about NovaChem: that people with somewhere better to go leave an employer that pays them less. A hiring rule built on that prediction punishes applicants for having alternatives, and turns the company’s own pay into a fault of theirs. A prediction can be accurate and still be no reason to act on it.

Gender, tested on its own, was clean. Most of the people in the overlap were women. They were a small part of all the women who applied, so their low scores barely moved the result for women as a whole. A test of one trait at a time could not see them. Inside the overlap the system marked down men and women alike. It scored the three lines, not anyone’s sex. Revathi had a reason, which she could not prove, for why most of the people with those lines were women. In many families she knew, sons were sent away to residential colleges, while daughters studied near home and took a further degree by post while they worked. Many legal systems call a rule that never mentions sex, and falls mostly on one sex, indirect discrimination, unless it can be justified.

A close relative of this pattern has a name. In 2018 four computer scientists described *fairness gerrymandering*, in which a classifier “appears to be fair on each individual group, but badly violates the fairness constraint on one or more structured subgroups defined over the protected attributes.”⁹ Their subgroups are defined by protected traits such as sex and race. Revathi’s was defined by ordinary lines on a CV, and most of the people in it were women. Reading the two as one pattern is mine. The law met a pattern like it earlier. Five Black women who had lost their jobs at a General Motors plant in St Louis sued for discrimination as Black women. In 1976 the federal court looked at race and at sex separately. It said they “should not be allowed to combine statutory remedies to create a new ‘super-remedy’”.¹⁰ The legal scholar Kimberlé Crenshaw took up that case in 1989, in the essay that named the problem *intersectionality*.¹¹ Fairness gerrymandering and intersectionality are not the same thing. One is a statistical failure of a classifier; the other is an account of how power falls on people who stand at more than one margin. Revathi’s case touches both. The vendor’s tests looked at one trait at a time, as most fairness testing has done.¹² Whether that is the reasonable method it looks like is a question this case raises and does not settle.

The print edition called Meera’s failure “a failure of her imagination”. Imagination sounds like a gift some people lack. An old Indian rule names something that can be asked of anyone. The *Ācārāṅga Sūtra* is the first of the twelve *aṅgas*, the basic scriptures of the Śvetāmbara Jain canon. Its first great vow, in Hermann Jacobi’s translation of 1884, is the vow of non-violence (*ahiṃsā*): “I renounce all killing of living beings, whether subtile or gross, whether movable or immovable.” It goes on: “Nor shall I myself kill living beings (nor cause others to do it, nor consent to it).” The vow names three ways of sharing in a harm: doing it, having it done and agreeing to it. The first clause of that vow is about walking.¹³

> A Nirgrantha is careful in his walk, not careless. The Kevalin assigns as the reason, that a Nirgrantha, careless in his walk, might (with his feet) hurt or displace or injure or kill living beings. Hence a Nirgrantha is careful in his walk, not careless in his walk.

A *nirgrantha* is a Jain monk, one “without bonds”; the *kevalin* is one who knows all; on this page the vows are taught by Mahāvīra. A later Jain text, the *Tattvārtha Sūtra* of Umāsvāti, is accepted by both of the tradition’s main schools. Written between the second and fifth centuries CE, it defines violence in one line: *pramattayogāt prāṇavyaparopaṇaṃ hiṃsā*. Manu Doshi translates it: “Deprival of life on account of non-vigilance constitutes violence.” His commentary adds: “Non-vigilance covers negligence, wrong motive, etc.”¹⁴ In the Digambara tradition the same line, read with Pūjyapāda’s commentary, is translated by S. A. Jain as “The severance of vitalities out of passion is injury”, and the commentary explains that “The person actuated by passion is *pramatta*.”¹⁵ The two translations pull apart on this chapter’s question. One finds the fault in not watching; the other in what moves the person who acts. Meera’s motive was good. Her watching was less careful than it could have been. Chapter 2 made a point that applies here. A harm nobody intended can still follow from an act somebody did intend. Meera did not intend the harm at the overlap. She did intend to sign. Read through the vow’s three ways, and the reading is mine: the vendor made the system, Basel had it made, and Meera consented when she signed.

The rule is for monks, and the beings it protects are insects on a path. I borrow only its order. Care is owed before you move, to beings too small for you to see, and the fault lies in not watching. *Pramāda*, carelessness, is a failure that anyone can be warned against. It can be named, its opposite asked for in advance, and the watching paid for.

By that measure, Meera walked as carefully as the vendor’s tests and her company’s forms asked in 2025, and less carefully than was already possible. The pattern had been published in 2018. Since July 2023 New York City has required employers who use automated hiring tools to publish audits that compare selection rates for “intersectional categories of sex, ethnicity, and race”.¹⁶ Its combinations are of sex, race and ethnicity only, not of traits like the three Revathi searched; but by 2023 testing combinations was already law somewhere. Vigilance has a standard, and the standard rises with what is known. Meera’s own history raised hers. She had already watched one screen turn away people nobody went back to.

It also has a limit. Nobody can test every combination of every trait. Someone has to choose which overlaps to look for, and that choice is not only statistical. Revathi looked at that overlap first because it looked like her own CV. That was a reason to look, not evidence of anything. And some traits cannot be tested without holding data that applicants might refuse to give and the law might forbid a company to keep, such as caste or religion. The audit could find only what the CVs showed. So the standard is proportionate: to how much the system decides, to what is already known, and to what looking would cost. It asks for care, not for knowledge of everything.

Designing for the people a system judges therefore has a practical side. The score can order the reading instead of ending it. Ordering has a cost too. Whoever the score puts last is read last, perhaps by a tired reader, and sometimes after the post is filled. A reader who sees a low score may also read with lower hopes. A score can be advice on paper and the decision in practice. The system can show its uncertainty, as a number with its record. It can flag the overlaps where its tests were thinnest. The print edition also offered a strategic reason: companies that look for overlooked talent find it. That promise is withdrawn. Sometimes they will, and sometimes the careful company will hire more slowly and pay more for it. The reason to design this way is owed to the applicants, not promised to the company.

## 31.4 Responsibility Engineered, Not Assumed

The title promises that responsibility can be engineered. What can be engineered is the seat: who may look, who may stop, who must reply and who is protected. The answering is still done by people.

The governance team was approved in November 2025 and began work in December. It has three members: a former quality analyst, Revathi, and a bioethicist. It reports to the site’s leadership and is accountable to the company’s global ethics board. From the start it could require the owner of any system to reply in writing to a finding and to test the system again. From the start, too, every review the team made of a new or changed system in Meera’s operations ended with her signature beside the owner’s. Systems outside her operations were signed by their own heads. The talent system was human resources’, so its owner at the site was the head of human resources. Meera’s signature of June 2025 came before the team existed. She had signed as the person leading the site’s AI work, while the system itself stayed with human resources.

The team’s terms gave it three further powers. It could suspend the site’s use of any system while it waited for a written reply. Only the site’s leadership could lift a suspension, in writing, with its reasons sent to the ethics board. Its members’ appraisals were written inside the team, not by the departments whose systems they tested. And its hours were budgeted, so that its work was not squeezed into evenings. Each was chosen to keep the seat full. Its holders could see the evidence and had the time and authority to refuse. They would not be punished for a good refusal, and they would have to explain.

Its limits were written down too. The team could stop the system at Hyderabad. It could not stop it at the company’s other sites. It could not decide whom to hire. Nor was it independent in every sense. Its members were NovaChem’s employees, paid from a budget the site’s leadership approved, and the leadership that could lift a suspension was also judged on how fast posts were filled. Its findings went to the ethics board in Basel as well as to the site, which was its protection and its limit. It could see every score and every CV, but not how the vendor had trained its model. The vendor’s documents described its tests, not what it had done with the company’s records.

Meera’s first wish, when she had read the eleven pages, was to stop the system at once. She could not. The stop was the team’s, and the system was not hers.

Revathi argued against stopping it. They were in Meera’s office with the door closed. “The human process it replaced had worse biases,” she said. “Before the system, hiring managers favoured their own colleges, their own networks, their own communities. The difference is that nobody could measure it. The system makes the bias visible. Removing the system doesn’t remove the bias. It makes the bias invisible again.”

Her sentence is half right, and the half matters. A human process can be measured. The study of Indian employers measured one. But nobody at NovaChem had measured its own until the team looked back, and the managers had kept only their shortlists, not a score for every applicant. On single traits, the team’s records showed the system’s shortlists were more even than the managers’ shortlists had been in the year before it. Stopping it would have stopped the harm in the overlap and returned hiring to a process that kept only its shortlists, with nothing to search. So the choice was not between a human and a machine. It was between a process whose decisions are recorded and checked, and one whose decisions are not.

“And every day we keep it, it marks them down,” Meera said. That was also half right. Keeping the system unchanged kept a harm that was now measured. The team did not suspend it. It chose to put the finding to the system’s owner. That choice had a price, and the team wrote it down: the system went on scoring for the days it took to arrange a meeting.

The meeting with the site’s head of human resources was in early August. He was a careful man. He asked how sure the finding was, and Revathi said what she could not prove. Six months of samples agreed. Whether the people in the overlap would have done well in the jobs was unknowable, because none of them had been hired. Meera told him what her own part had been. She had signed for the site’s use in June 2025 on tests of one trait at a time. This time she had asked what it measured, and accepted staying eighteen months with satisfactory reviews, though the old screen’s score had rested mainly on staying too. She had not asked about combinations, though researchers had been asking about them for years. That part was hers to answer for, and she said so before anyone asked. Then he asked whether the company was legally required to act.

It was not, in August 2026. At Hyderabad, where the scores were used on applicants in India, the European law did not reach that use at all, and no Indian law required such a test. The same system was also used for NovaChem’s plant near Lyon, in France. There the Act lists systems used “to analyse and filter job applications, and to evaluate candidates” among its high-risk uses, but those duties apply only from 2 December 2027.¹⁷ Even then, the duty to look for bias falls chiefly on the provider, and the article that sets it says nothing about combinations of traits. The EU’s data protection law already applied at Lyon. It limits rejections made by a machine alone, as the next section says. It asks nobody to test combinations of traits. Of the duties that will fall on NovaChem at Lyon as a deployer, one bears directly on a finding like Revathi’s. Suppose a deployer has reason to think that using a high-risk system as instructed may present a risk to health, safety or fundamental rights. It must tell the provider and the regulator, and “shall suspend the use of that system”.¹⁸ On a plain reading of that article, from December 2027 the law at Lyon would choose the stop that Revathi argued against. That would mean a return to a process that kept no score anyone could search. New York’s rule is a rare place where a test of combinations is required, and then only of sex, race and ethnicity. A state audit in December 2025 found that the city’s regulator had reviewed thirty-two companies and found one breach. The auditors found at least seventeen possible ones.¹⁹ A law that asks for the right test still needs someone to look.

Meera said: “We are not required. But we know. And once you know, the choice is no longer between acting and not acting. It is between acting and choosing not to act. Those are different things.”

He decided, within his authority, two things that morning. At Hyderabad the score would order the reading but would no longer reject anyone; every application would be read by a recruiter. That would take two recruiters’ time and slow hiring, and his budget would pay. With the first came two details, at Revathi’s request: the recruiters would read without seeing the score, and the team would record how far down the order each post’s reading had gone before it closed. The team would also record which version of the system had scored each applicant, because a finding is true of one version. And the governance team’s written finding would go to the global head of human resources in Basel and to the vendor, with a written reply and a new test required. A third thing, which Meera asked for, he took a day to decide.

Before they left, he asked Revathi to move into human resources to run the combined tests herself. She declined. “The person evaluating the system must not answer to the people who depend on it.” The principle is older and plainer than it sounds. Meera’s own industry has written it into its rules. In the European Union’s rules for making medicines, the heads of production and of quality control “must be independent from each other”.²⁰ NovaChem follows those rules. The person who checks a batch does not work for the person who needs it released.

Meera did not learn here that good intentions need structures. She had learned that earlier, and built some of them. The structures did not prevent this harm. They were the reason it was found.

## 31.5 AI and Inequality: Those Who Build and Those Who Are Scored

Before she showed anyone the table, Revathi had run her own CV through the test copy of the system. It had all three: the state university, the distance master’s and five years in IT services. The score came back in the lower third. She recorded it as one more case, and told Meera only this: “It would have rejected me a third time.” One case proves nothing, and she did not offer it as proof. It was a check on the instrument: a case where the system’s judgement could be compared with what had actually happened.

The print edition made this moment “the central inequality of the AI era” and called it a gap in consciousness. That is withdrawn. Nobody in this story was less conscious than anyone else. What differed was position. She could open the system, test it and argue with its owner. They could read a letter.

That is the inequality this section keeps. Systems like this one are built by people with education, money and access to data, and used on people without those things: applicants, borrowers, patients, people asking for benefits. The builders can examine a system. The people it judges see only its result. Each explanation of a single decision can be accurate while the pattern they make together is unfair, so a right to one explanation may not show the harm at all. A right that reaches only a person’s own data cannot reach the measure either. An applicant may want to ask why eighteen months of staying should decide her interview at all.

The rights to ask are thin and dated. In the European Union, the data protection law in force since 2018 already limits decisions made solely by machines. For such decisions it gives a person a right to meaningful information about the logic involved. From 2 August 2026 the AI Act adds, where EU law gives no such right already, a right for a person decided about by a listed high-risk system to “clear and meaningful explanations of the role of the AI system”, though it has little to hold until such systems are classed as high-risk in December 2027.²¹ Colorado’s law of May 2026, mostly in force from January 2027, will require a notice before use. After a decision that goes against someone, it will require an account of the system’s part in it, a chance to correct the record and, where commercially reasonable, a review by a person.²² India’s data protection law gives a person a right to “a summary of personal data which is being processed” and of the processing. That is not a right to an explanation, and most of the law’s duties apply only from May 2027.²³ India’s wage law forbids an employer to discriminate on the ground of sex when recruiting for the same or similar work. Whether that reaches a rule that never mentions sex but falls mostly on women is, as I read it, unsettled. A court will hear a charge under the wage law only if the government, a labour inspector, an employee or a registered trade union brings it. An applicant would have to persuade one of them to act.²³

There is another side to this, and honesty needs it. From July 2024, after the old screen and before the system, each recruiter at the site had hundreds of applications for every post. Each read the first page of most of them. For many applicants, the alternative to a score was never a careful human reader. It was a quick glance, or no reading at all. The machine read every page of every CV. A machine may yet be the first careful reader many applicants have had. Whether that weighs against the harm at the overlap depends on who is in the overlap, and this book does not settle it.

What did NovaChem owe the people it had already turned away? Five hundred applicants in the overlap had been rejected on the score in eleven months. The new rule would protect the next ones. It did nothing for them.

The philosopher Bernard Williams wrote about this in 1976, in an essay called “Moral Luck”. He imagined a lorry driver who, through no fault of his own, runs over a child. “We feel sorry for the driver, but that sentiment co-exists with, indeed presupposes, that there is something special about his relation to this happening, something which cannot merely be eliminated by the consideration that it was not his fault.”²⁴ Williams called what the driver feels *agent-regret*, a regret “which a person can feel only towards his own past actions (or, at most, actions in which he regards himself as a participant)”. The driver, Williams wrote, may do something he hopes will “constitute or at least symbolise some kind of recompense or restitution”, and that “will be an expression of his agent-regret”.²⁵ A spectator can be sorry. Only the driver can go to the family.

Here the two lamps disagree, and they answer different questions. The Jain texts ask when an act is violence. Their answer puts the fault in carelessness: the walker who watched the path has kept the vow, and the duty is in the walking. Williams asks what a person’s relation is to an outcome she did not intend. His answer is that care does not dissolve it. The driver without fault still stands in a special place, and he may offer a kind of recompense that no spectator would.

Each would let a builder off half the debt. The Jain rule has nothing to say once the walk is over. Williams has little to say about watching the path before it. Meera needed both. By the first, her fault was in how she had looked, and it could be named and paid for in the future. By the second, the people already turned away were hers, and the head of human resources’, and the vendor’s, in a way they were not a reader’s, whatever anyone’s fault. What she owed them was not a feeling. It was a search. On my reading, Williams describes a feeling and the acts that may express it, and stops before saying what is owed. The step from his regret to a debt is this book’s, and it rests on three things Williams does not say. Her signature made the system’s judgements partly hers. Her post added a duty to look. And the people it judged could still be found. Her share follows her power, what she knew and what she could still do. Regret may move a person to repair. It is not the repair.

Psychologists who study guilt and shame draw a line in the same place. Guilt, in their research, is about a particular act, and “corresponds with reparative actions including confessions, apologies, and undoing the consequences of the behavior.” Shame is about the whole self, and corresponds with attempts “to deny, hide, or escape”.²⁶ This describes what tends to happen. It is not advice to anyone in distress. But it fits what Meera asked for next.

The next day the head of human resources agreed to the third thing. Two recruiters would re-read every application in the overlap that the score had rejected, against the same written description of each job. The company would write to the people whose applications would have gone forward. It took them most of August. Of the five hundred, 123 would have been invited to an interview, about one in four. Most of the posts had been filled. Some had not, and others like them would open.

Whom to write to was the company’s own decision, and two questions met in it. Would a person have gone forward under a fair reading? And was she judged unfairly? Two recruiters could answer the first. The second was already answered for all five hundred. A remedy must also be safe to accept. A letter can offer a reading; the person decides whether to come back, and answering should never be held against her.

The site’s lawyers saw another side, and their worry points at a perverse incentive. A company that never looks has little on paper for a complaint to use. One that looks, and writes down what it found, has made the evidence itself. Where the law punishes finding a bias more surely than having one, it teaches companies not to watch. A law that wanted companies to look would treat those who audit, disclose and repair what they find more lightly than those who do not. A sentence saying that a harm was found is a statement of fact. It is not, by itself, an admission of legal fault, though lawyers fear it will be used as one.

By the same argument, the people the old screen turned away, the hundreds nobody went back to in 2024, were owed a search as well. Nothing in this story pays that debt. Nobody had asked to keep the old screen’s scores when it was removed, so by 2026 nobody could say exactly whom it had turned away. A debt can become unpayable when nobody keeps the ledger.

One piece of advice for the person on the wrong side of such a system comes with a warning. Ask what it measures: what does your screening system optimise for, and how does it handle career gaps? That is a good question. In India nobody is obliged to reply to it, and an applicant who asks may be marked as difficult. It is worth asking anyway, by people who can afford the cost. A right is worth what it costs to use.

## 31.6 The Builder’s Oath: A Proposal

Medicine’s oldest oath is sworn before witnesses. In Michael North’s translation for the US National Library of Medicine, it opens: “I swear by Apollo the physician, and Asclepius, and Hygieia and Panacea and all the gods and goddesses as my witnesses”.²⁷ Canadian engineers take an obligation written by Rudyard Kipling. Six of them first took it in Montreal on 25 April 1925, in a ceremony.²⁸ The print edition said that what the AI industry lacked was “the personal commitment ... the kind of promise a person makes to themselves, in their own voice”. An oath for data scientists already exists. In 2018 the US National Academies printed a “Data Science Oath”, modelled on a modern version of the Hippocratic Oath, which begins “I swear to fulfill, to the best of my ability and judgment, this covenant”.²⁹ Declarations exist too, such as the Toronto Declaration of 2018 on equality in machine learning, written by two human-rights groups, Amnesty International and Access Now.³⁰ What these texts lack is a second person: someone to whom the promise is made, who can ask about it later.

A promise, as this book has argued after the philosopher Hannah Arendt, is how people make a part of the future reliable to one another.³¹ A promise made only to yourself is a resolution. You can release yourself from it whenever you like, and nobody will know.

Regulations govern institutions. Oaths govern the self. But an oath governs the self only where someone else can ask the self about it. Who that someone is matters. A colleague can judge the work. Someone with authority can require action. Someone who speaks for the people judged can ask what they would ask. Each asks a different question.

So here is the oath this book proposed, in seven statements, revised. Each has been changed so that it promises only what the person who swears it has the power to do.

1. I will remember that the people my system judges did not choose it, and that no number I am asked to improve is a full account of them.
2. Before I build or sign, I will ask who gains and who bears the cost, and I will write down what I could not find out.
3. I will test for those my system is least likely to see, in combination, as far as the data allow. I will say which groups were too small or too sensitive to test.
4. I will say in writing what my system was tested for and what it was not. I will not sign for more, or hide what a test found.
5. I will look hardest at the edges, where my system was not designed to go and where, inside its design, it sees people least well, because harm gathers there.
6. I will speak when I see harm, in writing, to the person who can stop it, and where I hold power I will protect whoever speaks.
7. When my system wrongs people, I will help to find them and put right what can still be put right. What is beyond my power I will take to those who hold it. Being able to build a thing gives me no right to decide for them.

Two of the old statements changed most. The fourth promised “I will not allow my work to be deployed beyond its demonstrated capabilities.” Few builders hold that power; the deployer does. The seventh said that “my consciousness gives me the obligation to build well”. The obligation comes from what the system does to people, and that is now where the statement puts it.

### Words with history

*Responsible* came into English in the 1640s, through French, from the Latin verb *respondeo*. Lewis and Short’s Latin dictionary gives its first sense as “to promise a thing in return for something else”, and only then “to answer, reply”. Its root, *spondeo*, belonged to Roman law. The jurist Gaius writes that an obligation by words is made by question and answer: *dari spondes? spondeo*, “do you promise that it will be given? I promise” (my translation). The promise did not exist until someone else had asked for it.

In March 1788 *The Federalist*, no. 63, usually credited to James Madison, set two conditions on the English word. “Responsibility, in order to be reasonable, must be limited to objects within the power of the responsible party”. And “in order to be effectual”, it must relate to “operations of that power, of which a ready and proper judgment can be formed by the constituents”³¹. The first condition is why the seven statements promise only what their swearer can do. The second is why an oath needs a hearer able to judge it. The history proves nothing about what a builder owes. It shows that the word was made for a promise given to someone who may ask.

Oaths are not enforcement mechanisms. They cannot make a vendor test combinations or a budget pay for the testing. Who should pay for that testing? The legal scholar Guido Calabresi asked a question of accidents in general that fits it: who is best placed to weigh what failures cost against what it would cost to prevent them?³² Asked of this system, the question points to the vendor, which sells it to every site of the company and to other companies. The contracts in this story put the cost on the buyer anyway. Each hand’s share follows what it did and what it controlled. The vendor tested one trait at a time, and the measure was chosen in Basel. Meera signed on those tests, and the head of human resources owned the system at the site. What each paid is a separate question. Here his budget paid for the repair. Each hand swears its own oath. Meera cannot swear the vendor’s for it, or Basel’s. What an oath can do is name in advance what the swearer will be asked about, and by whom. Each hand can keep its own oath and the system can still fail, in the space between hands. That space needs an owner named before anything goes wrong, which is what the governance team was for. If a vendor charges extra for a test of combinations, a further question arises. For a system sold to decide about people, such a test is closer to a basic duty than to an extra. A buyer that keeps using such a system after learning of the gap, without asking for that test, takes on a larger share of what follows. And Hyderabad could find the harm and stop the rejections. Only Basel could change the measure, and only the vendor could change the model. Behind every hand stands the company, which outlasts them all. Meera may leave and the vendor may be sold; the debt to the five hundred stays with NovaChem, and so must the records that would let anyone pay it.

Some hands are a step further from the harm. A team whose platform lets its company do without testers does not decide who loses work. Its share is smaller than the share of the one who decides, but it is still a share. It owes a plain account of what its tool does and does not do, both to those who decide and to those the decision displaces. Its members may feel the regret Williams described without claiming a fault that was not theirs.

The aim, then, is not a system that is never unfair. Every system that judges people will be wrong about some of them, in ways nobody has yet looked for. The aim is a system whose unfairness is looked for, found, answered and not allowed to settle.

## Ending: A Letter to Madurai

The letters went out in the first week of September 2026.

There were a hundred and twenty-three. Each was signed by the head of human resources and offered either an interview for a post still open or a reading of the applicant’s CV for the next one. The site’s lawyers had drafted them to admit nothing. Meera had asked for one sentence to be added, and he had added it: “A computer program scored your application, and we have found that it scored some applicants unfairly, so a person has now read yours.”

The lawyers’ note was clipped to the master copy. The sentence, it said, could be used against the company in a complaint. He had read the note and kept the sentence. The re-reading had taken two recruiters most of a month, and the posts already filled could not be given back. Nobody wrote to the rest, nearly four hundred people. Their applications would not have gone forward even on a careful reading. The company decided that a letter would tell them only of a wrong that changed nothing for them.

The vendor had replied in late August. It could test combinations of traits, under a new contract, at a price. Basel had not yet answered.

Meera read one letter before the envelopes were sealed. It was addressed to a woman in Madurai who had applied in March for a post in quality control. The post was filled. The letter offered her the next one.

Revathi’s search was already running again, on a different set of traits. The system would be biased again. Not in the same way.

### Practice

*An oath someone can hold you to.* If you build, buy, approve or use a system that judges people, write the seven statements above in your own words. Strike any you cannot keep, and any that is not in your power, and beside each struck line write who does hold that power. Add the line this chapter’s case lacked: what you will do for the people already judged if the system turns out to be wrong, and who would find them. Then draw the map for one system you touch: who chose what it predicts, who signed for it, who can stop it, and who answers the people it judges. An empty box is where a harm will have no owner. Give the page to one person who sees your work and can ask you about it, and ask them to ask you, in six months, what you did. This takes an hour, and it may cost an awkward conversation. It will not make the system fair, and it promises no calm and no job. If instead a system has judged you, you can write to whoever used it and ask what it measures and how it treats a gap in a career. Nobody may be obliged to reply.

## Words to keep

- **agent-regret (Williams):** the regret a person can feel only about what he himself has done, even without fault, unlike a spectator’s.
- <strong><em>Ācārāṅga Sūtra</em></strong><strong>:</strong> the first of the twelve *aṅgas*, the basic scriptures of the Śvetāmbara Jain canon; its second book sets out the five great vows.
- <strong><em>ahiṃsā</em></strong><strong>:</strong> non-violence; in Jain teaching, the first great vow.
- <strong><em>aṅga</em></strong><strong>:</strong> literally “limb”; one of the twelve basic scriptures of the Śvetāmbara Jain canon.
- **Annex III:** the list in the EU’s AI Act of the areas where systems count as high-risk, including recruitment and the screening of job applications.
- **builder:** in this chapter, anyone whose choice shaped what a system does to the people it judges.
- **deployer and provider:** in the EU’s AI Act, the one who uses a system and the one who makes it.
- **fairness gerrymandering:** a pattern in which a system looks fair to each group taken alone and is unfair to a group defined by a combination of traits.
- <strong><em>hiṃsā</em></strong><strong>:</strong> violence; in the *Tattvārtha Sūtra*, the taking of life through non-vigilance.
- **indirect discrimination:** a rule that never mentions a protected trait, such as sex, but falls mostly on people who share it and cannot be justified; the United States calls the effect disparate impact.
- **intersectionality (Crenshaw):** the idea that a person can be harmed as a member of two groups at once in a way that neither group, looked at alone, shows.
- <strong><em>kevalin</em></strong><strong>:</strong> in Jain teaching, one who knows all.
- <strong><em>nirgrantha</em></strong><strong>:</strong> a Jain monk; literally, “without bonds”.
- <strong><em>pramāda</em></strong><strong>:</strong> carelessness, not watching.
- <strong><em>Tattvārtha Sūtra</em></strong><strong>:</strong> a Sanskrit summary of Jain teaching by Umāsvāti, written between the second and fifth centuries CE and accepted by both main Jain schools.

## Sources

1. Dennis F. Thompson, “Moral Responsibility of Public Officials: The Problem of Many Hands,” *American Political Science Review* 74, no. 4 (1980): 905–916; and “Designing Responsibility: The Problem of Many Hands in Complex Organizations,” in Jeroen van den Hoven, Seumas Miller and Thomas Pogge (eds.), *Designing in Ethics* (Cambridge: Cambridge University Press, 2017), pp. 32–56. Paraphrased. The provider and deployer: Regulation (EU) 2024/1689 (the Artificial Intelligence Act), Article 3(3) and 3(4).
2. Plato, *Republic* X, 601c–602a, trans. Benjamin Jowett (The Internet Classics Archive), paraphrased: the user knows the tool, the maker has right belief about it from the user, the imitator neither. That the scheme leaves no place for the person a tool is used on is this book’s reading.
3. Sukhadeo Thorat and Paul Attewell, “The Legacy of Social Exclusion: A Correspondence Study of Job Discrimination in India,” *Economic and Political Weekly* 42, no. 41 (2007): 4141–4145, findings: applicants with a Dalit name had odds of a positive outcome “0.67 of the odds of an equivalently qualified applicant with a high caste Hindu name”, and with a Muslim name “0.33”. Reported again in Sukhadeo Thorat, Paul Attewell and Firdaus Fatima Rizvi, *Urban Labour Market Discrimination* (New Delhi: Indian Institute of Dalit Studies, 2009), as summarised by the GSDRC (gsdrc.org): the odds of an invitation to interview were “about two-thirds” (Dalit) and “about one-third” (Muslim) of a high-caste Hindu applicant’s; no data were collected on who was finally hired.
4. Abhijit Banerjee, Marianne Bertrand, Saugato Datta and Sendhil Mullainathan, “Labor market discrimination in Delhi: Evidence from a field experiment,” *Journal of Comparative Economics* 37, no. 1 (2009): 14–27, abstract: no evidence of discrimination against non-upper-caste applicants for software jobs; “large and significant differences” in callbacks between upper castes and Other Backward Castes for call-centre jobs (the MIT DSpace version reads “larger”); “There is no evidence of discrimination against Muslims for either of the two kinds of jobs we apply for.” (RePEc, journal abstract).
5. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations,” *Science* 366, no. 6464 (2019): 447–453; the abstract puts the lower spending down to “unequal access to care”.
6. Jeffrey Dastin, “Amazon scraps secret AI recruiting tool that showed bias against women,” Reuters, 10 October 2018 (republished by Euronews the same day): “It penalized resumes that included the word ‘women’s,’ as in ‘women’s chess club captain.’” According to the people Reuters spoke to, the company had disbanded the team by the start of 2017, and its recruiters “never relied solely on those rankings”.
7. Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig and Sendhil Mullainathan, “The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables,” *Proceedings of KDD 2017*, pp. 275–284. Their case is bail; the application to hiring is mine.
8. Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, “Inherent Trade-Offs in the Fair Determination of Risk Scores,” in *8th Innovations in Theoretical Computer Science Conference (ITCS 2017)*, LIPIcs 67 (2017), 43:1–43:23; first posted as arXiv:1609.05807 (19 September 2016).
9. Michael Kearns, Seth Neel, Aaron Roth and Zhiwei Steven Wu, “Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness,” *Proceedings of the 35th International Conference on Machine Learning* (PMLR 80, 2018); abstract as in arXiv:1711.05144v5.
10. *DeGraffenreid v. General Motors Assembly Division*, 413 F. Supp. 142 (E.D. Mo., 4 May 1976), Wangelin J.; filed 1975 (Civ. A. No. 75-487 C (3)) (Justia); the plaintiffs had lost their jobs in the plant’s seniority-based layoffs of January 1974. Indirect discrimination: Directive 2006/54/EC, Article 2(1)(b): “where an apparently neutral provision, criterion or practice would put persons of one sex at a particular disadvantage compared with persons of the other sex, unless that provision, criterion or practice is objectively justified by a legitimate aim, and the means of achieving that aim are appropriate and necessary” (legislation.gov.uk); in the United Kingdom, the Equality Act 2010, s. 19, with the same exception where the rule is “a proportionate means of achieving a legitimate aim”; in the United States, “disparate impact”, 42 U.S.C. § 2000e-2(k) (law.cornell.edu).
11. Kimberlé Crenshaw, “Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics,” *University of Chicago Legal Forum* 1989, no. 1: 139–167.
12. Angelina Wang, Vikram V. Ramaswamy and Olga Russakovsky, “Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation,” *Proceedings of FAccT 2022* (DOI 10.1145/3531146.3533101), abstract: “Research in machine learning fairness has historically considered a single binary demographic attribute”; Kearns et al. (2018), abstract, likewise describe the prevalent definitions as fixing “a small collection of pre-defined groups”.
13. *Ācārāṅga Sūtra*, Book II, Lecture 15, “The Clauses”, the first great vow (p. 202) and its first clause (p. 203), in Hermann Jacobi (trans.), *Jaina Sūtras, Part I*, Sacred Books of the East 22 (Oxford: Clarendon Press, 1884) (sacred-texts.com). Jacobi’s spelling “subtile” is kept.
14. Umāsvāti, *Tattvārtha Sūtra* 7.8 (7.13 in the Digambara numbering), trans. and commentary Manu Doshi (Federation of Jain Associations in North America and Shrut Ratnakar, 2007), p. 122 (herenow4u.net; scan at jainqq.org). The application of a monk’s rule to a company’s hiring is mine.
15. S. A. Jain, *Reality: English Translation of Shri Pujyapada’s Sarvarthasiddhi* (Chennai: Jwalamalini Trust), p. 196, sūtra 13 of chapter VII (the Digambara numbering): “The severance of vitalities out of passion is injury”; the commentary: “The person actuated by passion is pramatta.” (jainqq.org scan).
16. New York City Department of Consumer and Worker Protection, Rules of the City of New York, Title 6, § 5-301(b), adopted 6 April 2023 under Local Law 144 of 2021: a bias audit must calculate selection rates and impact ratios separately for sex categories, race and ethnicity categories, and “intersectional categories of sex, ethnicity, and race”; enforcement from 5 July 2023 (Notice of Adoption, as reproduced by Sidley Austin; Ogletree Deakins, April 2023).
17. Regulation (EU) 2024/1689, Annex III, point 4(a) (“AI systems intended to be used for the recruitment or selection of natural persons ... to analyse and filter job applications, and to evaluate candidates”); Article 2(1)(c) (third-country deployers only where the output is used in the Union); Article 10(2)(f) and (g) (the provider’s examination for possible biases and measures to detect, prevent and mitigate them); all as in Regulation (EU) 2024/1689 as amended by Regulation (EU) 2026/1744 of 8 July 2026 (in force 27 July 2026), under which Annex III duties apply from 2 December 2027 (Article 113); Article 10(2)(f) and (g) are unchanged by the amendment (artificialintelligenceact.eu; AI Act Service Desk, consolidated text as at 27 July 2026).
18. Regulation (EU) 2024/1689, Article 26(5): where deployers “have reason to consider that the use of the high-risk AI system in accordance with the instructions may result in that AI system presenting a risk within the meaning of Article 79(1), they shall, without undue delay, inform the provider or distributor and the relevant market surveillance authority, and shall suspend the use of that system.” Article 79(1): AI systems presenting a risk are understood as a “product presenting a risk” as defined in Article 3, point 19, of Regulation (EU) 2019/1020, “in so far as they present risks to the health or safety, or to fundamental rights, of persons” (AI Act Service Desk, consolidated text as at 27 July 2026); that definition counts a risk only “to a degree which goes beyond that considered reasonable and acceptable”. “The regulator” in the text is the relevant market surveillance authority.
19. Office of the New York State Comptroller, “Enforcement of Local Law 144: Automated Employment Decision Tools,” Report 2024-N-6, 2 December 2025, covering July 2023 to June 2025: the city’s regulator reviewed 32 companies and found one instance of non-compliance; “We reviewed the same companies and identified at least 17 instances of potential non-compliance under LL144.”
20. EudraLex Volume 4, EU Guidelines for Good Manufacturing Practice, Part I, Chapter 2, “Personnel,” para. 2.5 (in operation from 16 February 2014): “The heads of Production and Quality Control must be independent from each other.”
21. Regulation (EU) 2024/1689, Article 86(1): a person subject to a decision taken by a deployer on the basis of the output of a high-risk system listed in Annex III (except point 2) has “the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken”; it applies from 2 August 2026, but systems are classed as high-risk under Annex III only from 2 December 2027. The earlier rights: Regulation (EU) 2016/679 (the General Data Protection Regulation), Articles 15(1)(h) and 22; under Article 86(3) of the AI Act, its own right applies only where Union law does not otherwise provide one. The GDPR’s Recital 71 gives “e-recruiting practices without any human intervention” as an example of a decision based solely on automated processing (gdpr-info.eu).
22. SB 26-189 (the Colorado ADMT Act), signed 14 May 2026, mostly effective 1 January 2027, repealing and replacing SB 24-205 (Morrison Foerster, 15 May 2026; Goodwin, June 2026), as described in Chapter 25.
23. Digital Personal Data Protection Act, 2023 (India), s. 11(1)(a): the right to obtain “a summary of personal data which is being processed by such Data Fiduciary and the processing activities undertaken by that Data Fiduciary with respect to such personal data”; sections 3 to 17 of the Act, including s. 11, commence on 13 May 2027 (notification of 13 November 2025, with the Digital Personal Data Protection Rules, 2025). India’s wage law: the Code on Wages, 2019, s. 3(2)(ii): no employer shall “make any discrimination on the ground of sex while recruiting any employee for the same work or work of similar nature”, except where the employment of women in such work is prohibited or restricted by law (s. 3(1), on wages, says “gender”); s. 52(1): a court takes cognizance of an offence under the Code only on a complaint by or under the authority of the appropriate government, an employee, a registered trade union or an Inspector-cum-Facilitator (advocatekhoj.com). These sections are in force from 21 November 2025 (S.O. 5322(E)), when the Code repealed the Equal Remuneration Act, 1976, whose s. 5 barred an employer recruiting for the same or similar work from making “any discrimination against women” (SCC Online, 26 November 2025; Indian Kanoon). The Supreme Court recognised indirect discrimination under the Constitution, in a case against the state, in *Lt. Col. Nitisha v. Union of India*, 25 March 2021 (Supreme Court Observer); as of 31 October 2026 I found no case applying it to a private employer’s recruitment under either Act.
24. Bernard Williams, “Moral Luck,” *Proceedings of the Aristotelian Society*, Supplementary Volume 50 (1976): 115–135; reprinted in Williams, *Moral Luck: Philosophical Papers 1973–1980* (Cambridge: Cambridge University Press, 1981), pp. 20–39, the lorry driver at pp. 27–28. Checked against the 1981 edition (the definition of agent-regret at p. 27, the lorry driver at p. 28); it agrees word for word with Dana K. Nelkin, “Moral Luck,” *Stanford Encyclopedia of Philosophy* (substantive revision 20 January 2025).
25. Williams, *Moral Luck* (1981), the definition of agent-regret at p. 27; “The lorry-driver may act in some way which he hopes will constitute or at least symbolise some kind of recompense or restitution, and this will be an expression of his agent-regret” at p. 28.
26. June Price Tangney, Jeff Stuewig and Debra J. Mashek, “Moral Emotions and Moral Behavior,” *Annual Review of Psychology* 58 (2007): 345–372, at 349–350.
27. The Hippocratic Oath, trans. Michael North (National Library of Medicine, 2002), as given on the NLM’s page on Greek medicine (nlm.nih.gov).
28. The Corporation of the Seven Wardens, “A Brief History of the Calling of an Engineer” (ironring.ca): Kipling’s texts of the Obligation and ceremony, approved in April 1925; first taken in Montreal on 25 April 1925 by six engineers.
29. National Academies of Sciences, Engineering, and Medicine, *Data Science for Undergraduates: Opportunities and Options* (Washington, DC: National Academies Press, 2018), Appendix D, Box D.2, “Data Science Oath”, p. 119, modelled on Louis Lasagna’s modern version of the Hippocratic Oath (1964), given in Box D.1.
30. Amnesty International and Access Now, *The Toronto Declaration: Protecting the rights to equality and non-discrimination in machine learning systems*, launched at RightsCon, Toronto, May 2018 (accessnow.org).
31. Hannah Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), §34, “Unpredictability and the Power of Promise,” paraphrased; see Chapter 4. Words with history (added with the feature in 31.6): Online Etymology Dictionary, “responsible” (etymonline.com: “1640s, from obsolete French responsible”, from Latin *respons-*, past-participle stem of *respondere*); Charlton T. Lewis and Charles Short, *A Latin Dictionary* (Oxford: Clarendon Press, 1879), s.v. *respondeo* (“to promise a thing in return for something else”; “To answer, reply, respond”) and *spondeo* (under early civil law a verbal contract was made by the question *spondes?* and the answer *spondeo*, citing Gaius, *Institutes* 3.91 ff.) (alatius.com edition); Gaius, *Institutes* III.92 (The Latin Library: “Verbis obligatio fit ex interrogatione et responsione, uelut DARI SPONDES? SPONDEO”), the translation in the text mine; *The Federalist*, no. 63, “The Senate Continued”, for the *Independent Journal* (The Avalon Project, Yale Law School, whose byline reads “Hamilton or Madison”), dated 1 March 1788 and attributed to James Madison in *The Founders’ Constitution*, vol. 1, ch. 13, doc. 33 (University of Chicago Press).
32. Guido Calabresi, “Optimal Deterrence and Accidents,” *Yale Law Journal* 84 (1975): 656–671, at 666, paraphrased; see Chapter 15.

*Named people in this chapter are composites, and their conversations and circumstances are constructed; the studies, laws and figures cited are real and sourced.*

## Revised from the print edition (2026)

- *What the print edition claimed:* that “optimisation is desire stripped of consciousness”; that the algorithm “has no desk. It has no eyes. It has processing, and processing is not seeing”; that the choice among fairness frameworks “cannot be computed. It must be judged. And judgement is an act of consciousness”; that asking what a system does to people “requires consciousness”; that a biased system has “no self to correct”; that an applicant who asks a question exercises “a form of consciousness that the system cannot”; that the central inequality of the AI era is “the consciousness gap”; that an oath’s force is “consciousness applied to power”, and that “my consciousness gives me the obligation to build well”; that “The machine does not ask” and that “Only a conscious being stands at the glass”; that moral responsibility lies “in that knowing”. It also claimed that the system “grew more confident” by learning its own outputs over eleven months; that Revathi’s shadow process hired the candidates the system would reject; that Revathi “called this epistemic closure”; that the AI Act “classified employment screening as high-risk” and that Meera “had spent months preparing NovaChem’s operations for compliance”; that the Act gives applicants “the legal right to ask”; that “India’s evolving data protection framework is building similar rights”; that companies which surface diverse talent find candidates narrow systems miss; that a non-linear career shows “the most valuable meta-skill in the AI era”; that the Toronto, Asilomar, IEEE and Montreal texts were the AI industry’s own declarations and that the personal commitment was missing; that independence was “a design principle for conscience”; and that Meera had “believed in culture”. It placed Meera in Jubilee Hills, awake under a ceiling fan; timed the meeting that chose the measure at “forty minutes”; told the adverse-event model as discounting “low-income countries”; had Revathi offered a permanent post and the revised model deployed the following quarter; spoke for “anyone from a non-dominant caste”; used “resume” for CV; and ended on a bridge to a teacher in Kyoto.
- *What I now hold:* a builder is anyone whose choice shaped what a system does to the people it judges, and the law splits the builder into provider and deployer. Knowing is where responsibility begins. A builder answers for what she looked for and failed to look for, and, once harm is found, for going back to the people it reached. Good intentions settle a motive and nothing about a debt. A measure is a theory of value, and the incentive of whoever chose it is written into it. A system judged only on those it lets through cannot learn its errors about those it turns away. A system can be fair on every trait taken alone and unfair where they meet, and a machine is good at finding where. Care is owed before acting and rises with what is known; repair is owed after a harm, whatever the fault, to everyone it reached, the old screen’s applicants included. A harm nobody intended can follow from an act somebody intended. A prediction true of a group can still be unfair to each person in it when the measure is narrow. A score that orders the reading still decides who is read last. A builder whose tool made a lay-off easy, though someone else decided it, still owes a plain account of what the tool does. What can be engineered is the seat, not the answering. A stop restores a process that kept only its shortlists; keeping a system unchanged keeps a measured harm. The inequality is one of position, not of consciousness. An oath governs the self only where someone else can ask about it, and it promises only what is in the swearer’s power. Causing a harm, being at fault, being liable in law and owing repair are four questions; a builder can owe repair without being most at fault. A prediction can be accurate and still be no reason to act on it, and a retention measure can turn an employer’s own pay into a fault of the applicants. A rule that falls mostly on women without mentioning sex is indirect discrimination, unless it can be justified. A score hidden from the reader orders attention less; a score shown to her can be advice on paper and the decision in practice. Whether a person would have gone forward and whether she was judged unfairly are different questions. Where the law punishes finding a bias more surely than having one, it teaches companies not to watch. A debt becomes unpayable when nobody keeps the ledger, and it stays with the company after every hand has gone. A test of combinations, for a system sold to decide about people, is closer to a basic duty than to an extra. The aim is not a system that is never unfair, but one whose unfairness is looked for, found, answered and not allowed to settle.
- *What changed my mind:* the Jain rule of the careful walk and the *Tattvārtha Sūtra*’s definition of violence as non-vigilance; Bernard Williams’s lorry driver; the 1976 court that would not let five women bring their case as Black women, and Crenshaw’s reading of it; the computer scientists’ account of fairness gerrymandering; studies of Indian employers that put the old bias at the desk; the dates of the EU’s law and New York’s rule, and the audit of New York’s enforcement; the problem of selective labels; this book’s earlier finding that nobody went back to the old screen’s applicants; research on guilt as a turn towards repair; three outside readings of an earlier version of this chapter, in October 2026; the full wording of the Jain first great vow (“nor cause others to do it, nor consent to it”); and India’s wage law on discrimination in recruitment.
- *What survives:* “Meera had approved the system herself.”; Revathi’s audit, clean on every trait and unfair at the overlap (“Each axis was clean. The intersection was not.” now told as the gender test clean while most of the overlap were women); “a theory of human value disguised as a technical specification”; “the difference between a mistake and an institution”; “Removing the system doesn’t remove the bias. It makes the bias invisible again.”, now half right; “We are not required. But we know.”; “The person evaluating the system must not answer to the people who depend on it.”; the applicant’s question about career gaps; “Regulations govern institutions. Oaths govern the self.”, with its condition; “Oaths are not enforcement mechanisms.”; the seven statements, revised; and “It would be biased again. Not in the same way.”
- *Section titles in the print edition:* 31.5 was “AI and Inequality: The Consciousness Gap Between Rich and Poor”. The other five keep their print titles.

*Next in the series:* [*Chapter 32: The Examined Life in the Age of Machines*](https://nithinpalal.com/ai/ai-proof/chapter-32-the-examined-life-in-the-age-of-machines)*.*