Evaluating the Effectiveness of Artificial Intelligence Systems in Intelligence Analysis

Daniel Ish’s report for the RAND Corporation, published in 2021, enters a debate that had been sharpening inside American intelligence for the better part of a decade — how to know, in disciplined terms, whether a machine-learning system deployed against an analytic problem is actually helping the analyst arrive at a better assessment. The question is not academic. By the time the report appeared, the intelligence community had spent years cycling through pilot projects, procurement drives, and interagency working groups aimed at bringing computer vision, natural-language processing, and pattern-recognition tools into the daily work of image interpretation, signals triage, open-source monitoring, and target development. Programme managers had grown fluent in the language of accuracy scores and confusion matrices, but the operational verdict — did the tool make the intelligence product better — often remained obscure long after the money had been spent. RAND is the natural venue for a report of this kind; the corporation has been advising the US defence and intelligence establishment since the earliest days of operations research, and its recent AI catalogue sits at the practical end of the field, closer to acquisition, test, and evaluation guidance than to grand strategy. Ish writes for the officials who have to sign off on systems whose outputs feed products consumed by policymakers, commanders, and operators.

The central argument runs against the tendency to treat evaluation as a technical afterthought bolted onto a delivery contract. Standard machine-learning yardsticks — precision, recall, F1, area under the curve — are necessary, Ish contends, but badly insufficient for the intelligence setting. A classifier can post strong numbers on a held-out test set and still degrade an analyst’s judgement in the field. It can anchor attention on the wrong cues; it can fail silently when the data it encounters no longer resembles the data it was trained on; it can produce outputs whose confidence is miscalibrated against the decisions the analyst is being asked to make. Effectiveness, in the report’s framing, is a property of the analyst-plus-system, not of the model on its own, and the evaluation regime has to be built accordingly. This is not a rejection of quantitative measurement — the report is emphatically for it — but a reordering: the numbers have to be produced against a task that resembles the mission, on data that resembles what will actually be seen, with human behaviour and downstream product quality kept in view rather than assumed away.

The report is organised as a walk through the layers at which evaluation has to happen, each layer answering a different question and requiring a different apparatus. At the component level, Ish covers the familiar territory of test-set metrics, but with attention to their pathologies. Imbalanced classes make raw accuracy meaningless; benchmarks drawn from convenience data flatter systems that would collapse on operational feeds; the independent-and-identically-distributed assumption on which most testing rests dissolves the moment an adversarial or denied environment intrudes. The section is a reminder to programme officers that vendor-supplied scores rarely travel intact from the test bed to the operations centre.

At the human-machine level, the report turns to a body of work on trust, over-reliance, and automation bias that has grown steadily since the first air-traffic and cockpit-automation studies of the 1980s. The concern here is that analysts confronted with an AI recommendation do not always integrate it in the way a designer intended: sometimes they defer to it more than the evidence warrants, sometimes they ignore it entirely, and sometimes they build informal workarounds that an evaluator never sees. Ish makes the case for instrumented workflows and controlled experiments that let evaluators observe not just what the system produced but what the analyst did with it. This is a harder kind of measurement — it requires ethnographic attention alongside statistical rigor — and the report is candid about how few programmes have the discipline or the resources to sustain it.

At the mission level, the questions become still less tractable. Does the system shorten the time to a finished assessment? Does it broaden the set of questions analysts can credibly answer? Does the product reaching the customer read differently, and better, than it did before? Ish acknowledges how rarely those outcomes are clean to measure — the causal chain from a tool’s introduction to a policymaker’s decision is long, and confounders are everywhere — but insists that the alternative, evaluating only what is easy to count, quietly incentivises the wrong systems. Some of the most useful adjustments to an analytic line, he observes, produce no shift at all in the local metrics an evaluator is watching.

Specific themes recur throughout the report and receive sustained treatment. Calibration of confidence is treated as a first-class concern. Intelligence products encode uncertainty in a controlled vocabulary — the words of estimative probability set out in ICD 203 and its predecessors — and a system whose numerical confidence scores do not translate cleanly into that vocabulary is going to distort the products it feeds. A model that reports 90 percent confidence when its historical hit rate at that threshold is 60 percent is not just wrong; it is systematically wrong in a way that pushes analysts to write more definite assessments than the evidence supports. The report walks through the calibration diagnostics that let evaluators catch this drift and argues for their inclusion in acceptance testing.

Distributional robustness gets similar attention. The intelligence problem is one in which the training set is almost always thin, skewed, and slightly out of date. Foreign-language analysis is the canonical example — the corpus that a Western contractor can assemble on a rare dialect or a niche jargon rarely resembles the material that will actually cross an analyst’s desk — and Ish uses it to argue for evaluation protocols that stress models against realistic distributional shifts before deployment rather than after. Rare-event detection is the other canonical hard case. A model tuned on a decade of quiet base rates will not automatically generalise to a period of upheaval, and the report presses evaluators to build in tests that anticipate that discontinuity.

Red-teaming and adversarial evaluation get a chapter’s worth of treatment on the premise that adversaries will probe deployed AI just as they probe human analysts. State-level opponents have every motive to learn how a Western intelligence system reasons and to feed it the inputs it handles worst. Poisoned training data, evasion attacks on deployed classifiers, and the exploitation of overfitted features are all discussed as things a test regime has to actively look for, not phenomena to be handled after they surface in the field. The report treats this as the natural extension of counter-deception tradecraft into the software layer, and it makes the point that the intelligence community’s culture of counterintelligence is an asset here that commercial evaluators do not have.

The data side of the problem runs through every chapter. Ground truth in intelligence is often delayed, contested, or simply unavailable — a target may not confirm itself for years, if ever — which makes the evaluator’s job structurally harder than in commercial machine learning. Ish returns repeatedly to the implications: labels are noisy, feedback loops are long, and the temptation to substitute a plausible proxy for actual mission outcome is constant. The report is unusually candid about the limits this places on quantitative evaluation and about the need for expert judgement — from senior analysts, from customers, from independent test authorities — to remain part of the assessment regime. It is not a call to abandon measurement; it is a call to keep measurement honest about what it can and cannot capture.

The examples the report draws on come from familiar terrain. Image classification against overhead imagery, where the labelled data is uneven and the operationally interesting cases are the rare ones. Text triage against open-source and signals intercepts, where language models trained on general corpora encounter jargon and adversarial code-switching. Anomaly detection against network and logistical data, where the definition of anomalous is itself contested. Named systems, doctrines, and programmes are handled with the reserve one expects from RAND — the report is not a survey of specific intelligence-community tools — but the operational textures are unmistakable to a reader who has followed the community’s public discussion of Maven and the various acquisition efforts that followed. The larger point is generalisable: the same evaluation questions that apply to a computer-vision system for imagery triage apply, with modifications, to a text classifier for tip-and-cue, to a link-analysis tool for target development, and to a forecasting system for indications and warning.

Reception has been what one would expect for a report of this kind: not a public event so much as a working document that has circulated through programme offices, test authorities, and academic centres focused on human-AI teaming. The report’s arguments align with a broader line of work that had already accumulated by the early 2020s — the growing literature on trustworthy machine learning, the AI test-and-evaluation guidance emerging from the Joint Artificial Intelligence Center and its successors, and the standards work at NIST that would eventually produce the AI Risk Management Framework. Where Ish pushes hardest is on the insufficiency of component-level metrics, and that position has strengthened rather than weakened as generative AI has forced evaluators to reckon with systems whose failure modes do not fit neatly into any confusion matrix. The counter-argument one hears in acquisition circles is essentially budgetary: the mission-level and human-machine evaluations the report calls for are expensive, slow, and organisationally demanding, and few programmes can sustain them at scale. Ish would concede the constraint but insist that the alternative — deploying systems whose real effectiveness is unknown — is the more expensive option once errors accumulate.

For a reader building a working library on AI in intelligence and defence, the report occupies a specific niche. It is not a strategy book and does not try to be one. It offers no prediction about who wins the next war or how machine learning reshapes the balance of power between the United States and its adversaries. It is a practical companion for the officials who have to translate any of those broader ambitions into a test plan, a request for proposals, or a memorandum for record. It pairs naturally with the human-AI teaming literature — Missy Cummings on automation and trust, Ben Shneiderman on human-centred AI — and with the growing shelf of official test-and-evaluation guidance from the Department of Defense and NIST. It does not cover the strategic questions that Paul Scharre, Kenneth Payne, and others have taken up, and it does not try to.

Some parts of the report will age faster than others. The specific benchmarks and datasets it references have already been overtaken by the shift from narrow classifiers to large foundation models, and any reader picking it up now will want to read its examples as illustrations rather than as current practice. The section on adversarial evaluation, written before the wave of prompt-injection and model-extraction work that followed the public release of large language models, is narrower than the corresponding literature is today. What is likely to age well is the report’s structural insistence that evaluation happen at three layers — component, human-machine, and mission — and that the community stop confusing strong scores on the first layer with effectiveness on the third. That framing is durable, and the operational reasons for it, if anything, have only sharpened as the systems reaching deployed users have grown more capable and less legible. For programme managers still trying to answer the founding question — is this working — the report remains a serious place to start.