Public policy analysis: why data-driven models often fail

That is the central error in much contemporary public policy analysis. Governments increasingly use data-driven systems to rank risk, forecast demand, detect fraud, allocate inspection capacity, prioritize patients, and direct police patrols. These systems can process volumes of information that no administrative team could handle manually. They can reduce delay. They can standardize routine decisions. In the OECD’s review of government AI cases, 57% of deployments supported automating, streamlining, or tailoring services, while 45% supported decision-making, sense-making, or forecasting.

The operational appeal is obvious. The analytical premise is less secure.

A predictive model answers a narrow question: given patterns in historical data, what is likely to occur next? Public policy must answer a harder one: what should government do, for whom, at what cost, and compared with what alternative? Those are not variants of the same question. They require different evidence, different public policy frameworks, and a different tolerance for error.

The failure begins when a forecast is treated as an intervention plan.

The fallacy of prediction as policy intervention

Prediction and causation are adjacent fields. They are not interchangeable.

A model may accurately identify households likely to miss a payment, neighborhoods likely to generate reported crime, or patients likely to incur high medical costs. None of this establishes that a particular policy response will improve outcomes. The first claim concerns correlation. The second concerns a counterfactual: what would have happened had the intervention not occurred?

That counterfactual is the engine of serious policy evaluation methods. If a city sends additional officers to a neighborhood identified as high-risk, reported incidents may rise because surveillance rises. If a health system assigns care-management resources to people predicted to be expensive, spending may decline without health improving. If a welfare agency flags applicants as potentially fraudulent, it may identify administrative irregularities while excluding eligible people who cannot navigate a more hostile process.

The dataset records an outcome. It does not automatically explain the mechanism that produced it.

This distinction is routinely flattened because predictive systems produce a number that looks decisive: a score, a rank, a probability. Administrative organizations like numbers because numbers can be routed through workflows. A probability above a threshold becomes a case review. A rank in the top decile becomes a service offer. A risk flag becomes a referral, an investigation, or a denial.

That conversion from score to action is where policy enters. It is also where the model’s apparent neutrality ends.

A public agency is not merely forecasting a social system. It is intervening in it. The intervention changes incentives, visibility, access, reporting behavior, and the composition of the next dataset. In systems terms, the model is part of the feedback loop it claims only to observe.

A forecast becomes a policy instrument the moment it changes who receives scrutiny, support, or public resources.

Evidence-based policy making is often invoked as if data volume resolves this problem. It does not. Better data can strengthen an analysis. It cannot eliminate the need to define the policy objective, identify causal pathways, test unintended effects, and decide which harms are unacceptable even when aggregate performance appears strong.

A model that predicts emergency admissions may be useful for staffing hospitals. A model used to determine who receives preventive care requires a different standard. The first is largely a capacity-planning problem. The second distributes a scarce benefit among people with unequal prior access to care. The technical architecture may look similar. The institutional stakes are not.

When proxies encode inequality: the health-care cost trap

The most instructive failures are often not failures of machine learning. They are failures of institutional measurement.

A 2019 study in Science examined a widely used population-health algorithm that identified patients for additional care. The system predicted future health-care costs and used that prediction as a proxy for health need. This choice appeared administratively rational. Costs are abundant in claims data. Need is harder to measure.

But health-care spending is not the same as illness.

For patients with the same predicted risk score, Black patients were found to be sicker than White patients. The system had learned an existing inequality in access and spending: lower expenditures among Black patients did not reliably indicate lower medical need. They could reflect differences in access to services, treatment patterns, trust, provider behavior, insurance structures, and accumulated barriers before a patient reaches the billing system.

When researchers replaced the cost proxy with a measure more closely aligned with health need, the estimated share of Black patients selected for additional help rose from 17.7% to 46.5%.

This is not a minor calibration issue. It exposes a structural problem in social impact assessment. Administrative data usually captures the activity of institutions more clearly than the condition of people affected by those institutions. Spending is visible. Unmet need is often not. Arrests are visible. Victimization that goes unreported is less visible. School attendance is visible. The pressures that prevent attendance may remain outside the data system entirely.

The proxy is therefore never just a technical shortcut. It is a policy decision embedded in a variable.

The usual response is to remove protected attributes such as race, gender, or disability status from the model. That can be appropriate. It is not a cure.

The system does not need an explicit racial field to reproduce racial disparities. Postal codes, prior expenditure, school history, contact with agencies, credit behavior, provider networks, and hundreds of other variables can function as proxies for unequal social conditions. Removing the label does not remove the structure that produced the data.

The harder task is conceptual rather than computational: determine whether the target variable represents the public objective at all.

A government that wants to improve health should model health, not merely expenditure. A government that wants to reduce harm should measure harm, not merely institutional contact. A government that wants to improve social mobility should not mistake short-term administrative compliance for long-term capability.

That requires qualitative knowledge. It requires frontline staff who understand how a benefit rule changes applicant behavior. It requires community organizations that can identify where an official dataset is blind. It requires affected people to be treated as sources of evidence rather than as residual error around a model.

Feedback loops and the illusion of targeted policing

Predictive policing is the clearest example of a model changing the world from which it learns.

Suppose an algorithm identifies a small set of blocks as locations with elevated crime risk. A police department deploys additional officers there. More officers observe more incidents, issue more citations, make more stops, and generate more records. The next training dataset now contains further evidence that those blocks are associated with enforcement activity. The model receives the signal as confirmation.

It is not necessarily confirmation of crime concentration. It may be confirmation of observation concentration.

Research on predictive policing has illustrated this mechanism through a simulation in which targeted deployment generated a 20% increase in crimes observed in targeted locations. The point is often misunderstood. The figure was not a documented 20% real-world increase in crime. It was an illustration of how additional observation, when fed back into a forecasting system, can make a model increasingly certain about the places it already targets.

The mechanism matters because public policy analysis too often treats recorded data as a passive description of reality. In many public systems, data is produced by institutional choices.

Police deployment affects recorded crime. Audit intensity affects detected tax noncompliance. Child-protection screening affects the count of recorded risk cases. Welfare fraud investigations affect the apparent distribution of fraud. Immigration enforcement affects the visible geography of undocumented residency. A dataset can be internally consistent and still be systematically shaped by prior enforcement decisions.

This creates a cascading effect:

1. Historical enforcement generates uneven records across locations and groups.

2. The model interprets those records as an underlying pattern of risk.

3. The institution deploys resources according to the predicted pattern.

4. Deployment produces more records in already-targeted places.

5. Those records are treated as new evidence that the original pattern was correct.

The model’s confidence rises. Its epistemic foundation may be weakening.

In a public system, an observed pattern may measure the state’s attention as much as it measures the public problem.

This does not mean every predictive policing system is invalid, nor does it establish that every crime dataset is irredeemably biased. It means that a model used in an adaptive social environment requires continuous examination of its data-generating process. Static validation is inadequate when the institution’s response alters the next observation.

The same principle applies well beyond policing. A fraud model that produces excessive false positives may deter legitimate claimants from applying. A child-welfare risk score may lead caseworkers to document more intensively in families already subject to scrutiny. A hospital triage tool may change referral patterns and thereby alter the prevalence of recorded illness among the patients it sees.

The model is no longer outside the system. It is one of the system’s causal inputs.

Accuracy is not the governing metric

The political temptation is to demand that a model be “accurate.” The analytical problem is that accuracy is an incomplete standard.

A model can be accurate overall while failing the people for whom an error is most costly. It can be statistically well-calibrated but operationally destructive. It can outperform human judgment on one metric while undermining procedural fairness, public trust, privacy, or legal rights. It can be accurate in the historical sample and unreliable after a policy change, economic shock, demographic shift, or modification to agency practice.

The appropriate question is not whether an algorithm beats a baseline score. It is whether the whole decision system improves the public objective without creating unacceptable harms.

That system includes at least five layers:

• The policy objective. Is the institution seeking to predict a future event, prevent an event, allocate a benefit, ration a scarce resource, or detect misconduct? These goals demand different forms of evidence.
• The target variable. Does the outcome being predicted correspond to the social condition government actually intends to improve, or is it merely a convenient proxy?
• The decision rule. What happens after a score is produced? Is it an advisory signal, an automatic denial, a priority ranking, or a trigger for human review?
• The operating environment. Does use of the model alter behavior, reporting, access, or enforcement patterns in ways that corrupt future data?
• The remedy. Can a person understand, challenge, and correct a decision that affects their housing, benefits, health care, education, liberty, or livelihood?

This is why public policy frameworks cannot be imported directly from consumer technology. A recommendation engine that displays an irrelevant product has produced a low-stakes error. A public model that wrongly identifies a family as high risk, withholds medical support, or channels intensified surveillance toward a neighborhood has activated state power.

The error rate is only one component of the harm function.

The National Institute of Standards and Technology frames trustworthy AI through a broader set of characteristics: validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. This is closer to the right structure, though it remains a voluntary risk-management framework rather than a binding rule.

The U.S. Government Accountability Office organizes accountability around four practical principles: governance, data, performance, and monitoring. That sequence is revealing. Performance is not first. It sits within a wider operating model.

A system with weak governance cannot be rescued by a high-performing classifier. A system trained on unrepresentative data cannot be repaired by elegant reporting. A system that is never monitored will eventually drift from the population and conditions that produced its original performance.

The validation gap in modern government systems

Validation is often treated as the final technical gate before deployment. In public administration, it should be a continuing institutional discipline.

The United Kingdom’s updated AQuA Book defines validation as testing whether an analytical product meets user requirements. It identifies measures such as sensitivity, specificity, accuracy, precision, and reproducibility, while also requiring validation of both the conceptual model and any computerized implementation.

The conceptual model deserves more attention than it receives.

A computerized model can be flawlessly implemented and still answer the wrong question. Its code can be reproducible. Its data pipeline can be secure. Its predictive metrics can be strong. Yet the system can remain invalid because its target variable is a poor proxy, its training population differs from the current population, or its policy use rests on an untested causal assumption.

A serious validation regime therefore needs to run at several levels.

Validate the policy theory before validating the software

Every model carries an implicit theory of change. If a city ranks properties by fire risk, the theory may be that inspections reduce fire harm. If an agency ranks jobseekers by unemployment risk, the theory may be that early intervention improves employment outcomes. Those propositions should be tested directly where possible.

The key question is not, “Can the system identify high-risk cases?” It is, “Does acting on this ranking produce better outcomes than plausible alternatives?”

That requires comparative evaluation: randomized trials where appropriate, quasi-experimental designs where trials are infeasible, phased rollouts, matched comparison groups, and pre-specified measures of both benefit and harm. Prediction may form part of the operational tool. It cannot substitute for evaluation of the intervention.

Audit the data as an institutional record

Data quality is not limited to missing values and duplicate records. In government systems, quality also includes representativeness, provenance, and the administrative conditions under which a record was created.

Agencies should ask:

• Which groups are underrepresented because they have weaker access to services, lower reporting rates, language barriers, or distrust of public institutions?
• Which variables record human need, and which mainly record previous bureaucratic contact?
• What policy changes, enforcement campaigns, or eligibility shifts altered the data series?
• Are data labels based on verified outcomes, discretionary judgments, or institutional actions that vary by office and geography?
• Has the model been trained on a period whose economic or demographic conditions no longer hold?

These are not peripheral questions for an ethics committee. They determine whether the model has a stable object to predict.

Test subgroup performance and decision consequences

Aggregate accuracy can conceal concentrated failure. A model should be tested across relevant populations and contexts, but the analysis cannot stop at subgroup error rates. The agency must also examine what the error does.

A false positive in a system that offers voluntary support may be inconvenient. A false positive in a system that triggers investigation, benefit suspension, or law-enforcement contact may be severe. Similarly, false negatives can mean missed preventive care, unaddressed housing risk, or delayed intervention.

Thresholds should therefore be tied to consequences, not selected because they produce a clean operational workload. An agency may have capacity to review only 5% of cases. That capacity constraint is real. It should not be disguised as a neutral risk threshold generated by the model.

Monitor after deployment, not only before it

Models decay. Public systems change faster than their documentation suggests.

A new benefit rule alters claimant behavior. A recession changes labor-market patterns. A hospital merger changes referral flows. A policing strategy changes where incidents are recorded. The model’s inputs may retain the same names while their social meaning shifts underneath them.

Monitoring must therefore cover more than technical drift. It should track:

• changes in the composition of people subject to the model;
• changes in referral, approval, denial, and escalation rates;
• outcome disparities across affected groups;
• evidence that the model is altering the availability or quality of its own data;
• complaints, appeals, overrides, and frontline reports that reveal failures invisible in aggregate dashboards.

The last category is regularly undervalued. Human overrides are not merely evidence of staff resistance to modernization. They can reveal model blind spots, broken workflows, and conditions absent from administrative data. If every override is coded as noncompliance, the institution removes one of its few channels for learning about model failure.

Human judgment is not the alternative to rigor

The debate is often framed as a false choice between automated systems and subjective human judgment. That framing is useful for vendors and unhelpful for policy.

Human decision-making is inconsistent. It can be biased, poorly documented, slow, and vulnerable to organizational pressure. Public institutions should not preserve opaque discretion merely because algorithmic systems have limits.

But the correct alternative is not to automate discretion without examining the public policy assumptions embedded in the automation. It is to build decision systems in which statistical tools handle tasks they can genuinely improve, while human judgment is structured, accountable, and informed by evidence the model cannot capture.

This means reserving automated decisions for narrower contexts with bounded consequences, stable objectives, high-quality data, and clear routes for correction. It means using models as triage tools rather than as final arbiters where rights or essential services are at stake. It means publishing enough about objectives, data sources, validation methods, and monitoring results for outside scrutiny to be possible.

The 2025 U.S. federal guidance on AI adoption emphasizes safeguards for privacy, civil rights, civil liberties, and the mitigation of unlawful discrimination. That direction is necessary. But guidance alone does not create implementation capacity. Governments still face skills gaps, fragmented data systems, weak procurement oversight, and institutional incentives that reward deployment announcements more than long-term evaluation.

The structural friction is straightforward: agencies are asked to modernize quickly, but responsible deployment is slow. It requires governance, domain expertise, legal review, data stewardship, frontline participation, and recurring evaluation. A dashboard is cheaper than that infrastructure. It is also far less likely to detect when a model has mistaken administrative traces for social reality.

The real standard for public policy analysis

Data-driven models do not often fail because they are insufficiently sophisticated. They fail because institutions ask them to settle questions that are political, causal, and moral in structure.

No model can determine how society should balance false accusations against missed cases, efficiency against due process, or targeted support against universal provision. Those are public choices. Analytics can clarify their likely consequences. It cannot make them disappear.

The strongest use of data in public policy analysis is therefore not automated certainty. It is disciplined uncertainty: a system that states what it predicts, what it cannot infer, whose experience it may fail to represent, and how its decisions will be tested once they meet the real world.

That is less theatrical than the promise of algorithmic government. It is also how public institutions avoid turning historical inequality, administrative convenience, and unexamined assumptions into a feedback loop with official authority.