Awkward Intelligence

Katharina Zweig’s Awkward Intelligence arrived in English in 2022 as a translation of her 2019 German bestseller Ein Algorithmus hat kein Taktgefühl — literally, an algorithm has no sense of tact. The move from a German trade imprint to MIT Press reframed the book: what had been a popular guide for Berlin’s policy debate became an entry in the English-language canon on algorithmic accountability, published while the EU AI Act was being negotiated and American scholars were still arguing over the fallout from the COMPAS recidivism scandal. Zweig writes from TU Kaiserslautern, where she built Germany’s first degree programme in Socioinformatics, and from a seat on the Bundestag’s Enquete Commission on Artificial Intelligence. She is also one of the co-founders of AlgorithmWatch, the Berlin-based NGO that spent much of the previous decade auditing Schufa credit scores and the Austrian employment agency’s job-seeker ranking system. The book carries that provenance throughout. It is not a computer scientist describing her field to the public; it is a computer scientist who has spent years in the room where regulators try to translate her field into law, writing for the people who will have to live under the resulting rules.

The central argument is deceptively plain. An algorithm, Zweig insists, is never purely a piece of mathematics. Every system that makes or supports a decision is a socio-technical construct, and the human choices embedded in it begin long before any line of code is written. The question that matters is not whether a given task can be automated — the answer is almost always yes, in some technical sense — but whether it should be, and under what conditions, with what oversight, and with what route back when it fails. Zweig’s position is neither cheerleading nor refusenik. She is impatient with the industry’s habit of presenting machine learning as a neutral mirror on reality, and equally impatient with the cultural pessimism that treats every algorithm as a Kafkaesque imposition. What she wants is a vocabulary clear enough that a parliamentarian, a judge, or a hospital administrator can tell the difference between a system that is awkward and one that is dangerous, and know which questions to ask before signing the procurement contract.

To make that vocabulary portable, the book is organised around a model Zweig calls OMA — Operationalisation, Modelling, and Algorithmicising. The idea is that any claim to automate a human judgement passes through three distinct steps, each of which smuggles in choices that are not technical at all. Operationalisation is the move from a social concept to a measurable proxy: deciding, for instance, that “recidivism” means being rearrested within two years, rather than reconvicted, or reoffending in some unobserved sense. Modelling is the move from the proxy to a set of input features: which facts about a defendant will be allowed to count. Algorithmicising is the move from features to a fitted model that outputs a score. Each stage is a translation, and each translation loses something. Zweig’s point is that the losses are not evenly distributed, and are rarely visible to the people the system acts upon. By the time a judge sees a COMPAS risk score, the questions “what counts as risk?” and “whose history counts as evidence?” have already been answered on their behalf by people they will never meet.

The book is built in roughly four movements. The first lays out why “intelligence” is the wrong frame for what current systems do, drawing a careful line between pattern recognition and anything that resembles judgement. Zweig walks through supervised learning in enough detail for a non-specialist to follow, using worked examples — Netflix’s recommender, a breast-cancer classifier, a spam filter — to show both what these systems are good at and why their outputs come with error rates that cannot be waved away. She returns often to a simple statistical point that most public coverage still gets wrong: a classifier with a 95 percent accuracy rate on a rare condition can still produce vastly more false positives than true ones, and the arithmetic matters more than the headline figure. The chapter on base rates and confusion matrices is the book’s most durable technical contribution, because it gives a lay reader a tool they can use on any system they encounter in the news.

The second movement moves from the mathematics to the social machinery around it. Here Zweig introduces her typology of algorithmic decision-making systems, ranked by the stakes involved and the degree of human oversight. A film-recommendation engine sits at one end: the cost of a wrong suggestion is a wasted evening, and the user corrects it by watching something else. A pretrial risk-assessment tool sits at the other: the cost of a wrong classification is a person held in jail who should not be, or released who should not have been, and the person affected has almost no feedback channel. Between those poles she places credit scoring, employment screening, child-protection triage, and medical pre-diagnosis. The book argues that the regulatory conversation has often conflated these very different regimes under a single term — “AI” — and that the first task of any serious policy is to disentangle them. Her proposed matrix, cross-tabulating potential harm against reversibility, became one of the templates for the risk-tiered approach that eventually surfaced in the EU AI Act.

The third movement is the one most readers remember. Zweig spends several chapters on the COMPAS case, in which the US company Northpointe (later Equivant) sold a proprietary recidivism score to counties across the United States, and ProPublica’s 2016 investigation found that the system misclassified black defendants as high-risk at roughly twice the rate it misclassified white defendants. She reconstructs the ensuing fight between ProPublica’s journalists and Northpointe’s statisticians patiently, and uses it to introduce the impossibility results of Alexandra Chouldechova and Jon Kleinberg: that under any realistic data distribution, several intuitively desirable notions of statistical fairness cannot all hold at once. You can calibrate your risk scores so that a “high-risk” label means the same probability of reoffending for everyone, or you can equalise false-positive rates across groups, but you cannot do both. Zweig does not treat this as a defence of COMPAS; she treats it as a reason to argue that the choice between fairness definitions is a political act, not a technical one, and that any vendor that pretends otherwise is selling you a question dressed up as an answer. The chapter has been cited in German case law and in several European procurement guides.

Other case studies populate the same movement. Microsoft’s chatbot Tay — released on Twitter in 2016 and withdrawn within sixteen hours after users taught it to deny the Holocaust — serves as her example of what happens when a learning system meets a hostile input distribution and has no notion of social context. Schufa, the German credit bureau whose scoring formula remains a trade secret despite affecting almost every tenancy and loan in the country, is the long-running antagonist of the AlgorithmWatch work she helped lead. The Austrian AMS algorithm, which classified unemployed job-seekers into three categories and allocated retraining budgets accordingly, gets a close read: Zweig walks through the input features — age, nationality, caregiving duties — and shows how a system intended to optimise return-to-work rates ended up directing resources away from the groups that most needed support, because the ground-truth label was defined by the current labour market rather than by any notion of what a fair labour market would look like. Each case is used to illustrate one specific point about the OMA pipeline, so the examples accumulate into an argument rather than a parade.

The fourth movement is the most hopeful part of the book and the easiest to miss, because it reads at first like a checklist. Zweig lays out what she calls the long transparency chain — the set of questions that any auditor, journalist, regulator, or affected citizen should be able to answer about a deployed system. What was operationalised, and by whom? What features entered the model, and what was deliberately excluded? What error rates does the vendor claim, on what test set, and have those been reproduced? Who gets a decision, and who gets a decision explained? What is the appeal path, and how long does it take? She is explicit that no vendor will volunteer this information, and no procurement contract she has seen asks for it in a form that permits comparison across suppliers. The chapter ends with a model clause for public tenders, cribbed from the German parliamentary report she co-authored, that specifies minimum disclosure in a way that does not require the buyer to reveal the vendor’s trade secrets. It is one of the few places in the recent AI-ethics literature where the writer offers something a civil servant can lift directly into a document.

Zweig closes with a short meditation on why “awkward” is the right word. Her machines are not evil; they are not conscious; they are not the superintelligent agents of the longtermist imagination. They are tools that trip over social complexity because they were built to optimise a measurable proxy for something that was never fully measurable in the first place. The appropriate response is not fear or veneration but a slower, more unglamorous attention — the kind that insists on asking what the error rates look like for the people whose names will never make the newspaper. The posture is European in a specific sense. It is comfortable with regulation, sceptical of the Silicon Valley self-image, and uninterested in either utopia or apocalypse. Readers coming from American tech-criticism will recognise many of the destinations and few of the stops along the way.

The book’s reception has been strongest in exactly the quarters that shaped it. The German edition sold through several reprints, and Zweig became a regular witness in Bundestag hearings during the drafting of the AI Act. English-language reviewers at the LSE Review of Books and the Journal of Information Policy praised the clarity of the OMA framework and the restraint of the voice. Critics have pointed out two gaps. The book was finished before the public release of ChatGPT in late 2022, so the generative-AI boom and the specific problems of large language models — hallucination, training-data provenance, agentic use — fall outside its scope. Readers who come to it hoping for a guide to the current moment have to extend the framework themselves, which is possible but not always easy; a classifier with a visible confusion matrix is a different animal from a chatbot whose errors arrive as prose. The second gap is geographical. Zweig writes from a German regulatory tradition, and her examples are weighted toward Europe and the United States. The treatment of Chinese social-credit systems is brief, and the question of military applications — autonomous targeting, battlefield decision-support — is left aside almost entirely. For a reader interested in those domains the book is a methodological foundation, not a destination.

What role it plays, then, depends on where the reader is standing. For a civil servant or a parliamentarian trying to vet a procurement proposal, it is one of the few books that provides vocabulary and a checklist in roughly equal measure. For a journalist covering an algorithmic scandal, it is a reference text for base rates, fairness definitions, and the limits of what any vendor’s “accuracy” number can mean. For a working data scientist it is a usefully humbling tour through the social afterlife of their own decisions. It pairs well with Virginia Eubanks’ Automating Inequality on the human costs of welfare automation, with Cathy O’Neil’s Weapons of Math Destruction on the political economy of scoring systems, and with Michael Kearns and Aaron Roth’s The Ethical Algorithm on the technical side of fairness constraints. It does not pair well with any book that promises to tell you what artificial intelligence is going to do next, because that is not the register Zweig writes in.

What is likely to age well is the OMA framework itself and the insistence that each of its three stages is a site of political choice. Those ideas will survive the next wave of models and the one after, because they describe something true about the handover from a social concept to a technical one, and that handover is not going to get any smaller. What has already begun to date are some of the specific case studies — Tay feels like a period piece, Schufa is still Schufa — and the implicit model of what an AI system looks like. The next edition will need a chapter on foundation models, and probably one on the military uses that this one declines to touch. The core instinct, that the right question is not whether a machine can decide but under what conditions a human should let it, has if anything grown more useful as the systems have grown larger and the oversight has grown thinner.