Must We Understand a Treatment to Trust It?

In his talk Math 2.0, Terence Tao presents a thought experiment about an AI-generated cancer treatment. The AI supplies a Lean certificate for its prediction, and the treatment passes a phase III clinical trial. Tao asks whether a patient would find it reassuring to know that at least one human expert understands, even partially, the treatment’s mechanism or the mathematical model behind it. [1]

For many people, the answer is obviously “no.” Knowing that a human expert partially understands the mechanism would add little or nothing to their reassurance. They would want convincing evidence that the treatment works, that its risks have been investigated, and that it is appropriate for their condition. Whether somebody can explain how it works is a different question.

Human beings have relied on useful empirical knowledge for millennia without understanding the underlying mechanisms. People learned that apples provide nourishment and sand does not, long before they knew about vitamins, metabolism, or digestion at the molecular level. Their confidence came from experience. An explanation would have enriched their knowledge, but its absence gave them no reason to stop eating apples.

Penicillin provides a more directly relevant example. Its therapeutic use began in the early 1940s, whereas the identification of its inhibition of a particular cross-linking reaction in bacterial cell-wall synthesis came in 1965. Doctors knew that it attacked susceptible bacteria well before they understood the detailed molecular mechanism. Patients benefited during that interval; treatment did not have to await the explanation. [2], [3]

The distinction also appears in our use of technology. Semiconductor devices depend on quantum physics, yet a person can reasonably use a mobile phone without understanding that physics or caring about disputes over its interpretation. Engineers understand semiconductor behavior extremely well, so this example does not establish that expert understanding is dispensable. It does show that practical confidence need not depend on personal understanding or on resolving every foundational question. [4]

There is another issue: can a machine deserve greater trust than a human? In particular tasks, the answer is plainly yes. Someone may step outside and estimate that the temperature is 90°F. A properly functioning, calibrated thermometer will ordinarily provide a more accurate measurement. We do not distrust its reading because the instrument has no understanding of temperature. We trust it because its performance has been established.

Aviation provides a more consequential example. Commercial aircraft equipped with certified automatic landing systems can land in visibility conditions that do not permit a manual landing. Pilots supervise these operations, but the precise execution of the landing is entrusted to automation. A passenger could reasonably prefer that arrangement to an attempt to land manually under the same conditions. Airbus explicitly describes autoland as enabling landings in poor visibility when a pilot cannot perform a manual landing. [5]

Autonomous driving offers further evidence. A study involving Waymo and the insurer Swiss Re reported substantially lower liability-claim rates for Waymo’s driverless service than for geographically matched human-driver benchmarks. These findings concern a particular system and operating conditions; they do not establish that every self-driving car is safer everywhere. Nevertheless, they support the possibility that choosing software over a human driver can already be a rational safety decision. [6]

Neither thermometers nor automatic landing systems reproduce the precise situation in Tao’s thought experiment: their operation is understood by human designers. Their relevance is narrower but important. They undermine any presumption that human involvement is inherently more trustworthy. Trust should depend on demonstrated reliability for the task at hand. If an AI-generated treatment were supported by better evidence than a human-designed alternative, its origin would provide no sufficient reason to prefer the alternative.

Tao’s formulation also sets a surprisingly weak standard: at least one human expert who understands the mechanism even partially. How much reassurance should that provide? An expert might correctly understand one component while overlooking a dangerous interaction. A mathematician might thoroughly understand a model whose assumptions fail in patients. Partial understanding can be useful, but the mere existence of someone possessing it is hardly a compelling assurance of safety.

Tao’s broader concern deserves to be distinguished from his question about reassurance. The surrounding slides make clear that he is concerned about powerful optimization exploiting a gap between a measurable target and the outcome we actually want. He explicitly acknowledges that a lack of human understanding is not inherently dangerous. [1] The examples of apples, penicillin, and thermometers do not resolve the optimization problem. They establish a narrower point: reliable practical knowledge does not always require a mechanistic explanation.

The concern about exploiting weaknesses in clinical trials is legitimate. Passing a trial does not establish every desirable property of a treatment, and a Lean certificate establishes a formal claim under specified assumptions; it cannot, by itself, establish that those assumptions adequately describe human biology. But human understanding offers no automatic escape from these difficulties. A persuasive mechanistic explanation can coexist with a misleading model or inadequate evidence.

Identifying a possible failure of empirical evaluation does not establish that a human’s partial understanding would detect or prevent it. To make that argument, we would need to explain what the expert understands, which failure that understanding could reveal, and whether the resulting safeguard is more reliable than additional independent testing. Many patients would reasonably care about those concrete contributions while finding little reassurance in the bare statement that somebody partially understands the treatment.

The relevant question is whether the treatment has survived sufficiently demanding, independent empirical scrutiny. Understanding its mechanism may help researchers design that scrutiny, identify risks, and determine where the treatment will fail. Those are concrete benefits. They should be assessed as such, rather than assumed merely because a human can offer an explanation.

If the evidence were otherwise identical, some patients would certainly feel reassured by a human expert’s partial understanding. Others would attach negligible weight to it. Many would quite reasonably prefer a poorly understood treatment with strong evidence of benefit over an elegantly explained treatment with weaker evidence.

Understanding remains a major scientific achievement. It can make treatments safer, suggest improvements, and open further avenues of research. But its scientific value does not establish that every patient needs it as an additional source of reassurance. Tao’s thought experiment invites an intuition that many people simply do not share.

References

1. Terence Tao, Math 2.0, public lecture at Caltech, October 9, 2026. The thought experiment appears on slides 20–21; its context appears on slides 19 and 22–24.
   Lecture slides (PDF).
2. American Chemical Society, “Discovery and Development of Penicillin,” National Historic Chemical Landmarks.
   Historical account.
3. E. M. Wise, Jr., and J. T. Park, “Penicillin: Its Basic Site of Action as an Inhibitor of a Peptide Cross-Linking Reaction in Cell Wall Mucopeptide Synthesis,” Proceedings of the National Academy of Sciences 54 (1965), 75–81.
   https://doi.org/10.1073/pnas.54.1.75.
4. Jing Wang, Eric Polizzi, and Mark Lundstrom, “A Three-Dimensional Quantum Simulation of Silicon Nanowire Transistors with the Effective-Mass Approximation” (2004).
   https://arxiv.org/abs/cond-mat/0403739.
5. Airbus, “Flight Test – Autoland Tests,” February 28, 2023.
   Airbus account of automatic landing and certification.
6. Luigi Di Lillo et al., “Comparative Safety Performance of Autonomous- and Human Drivers: A Real-World Case Study of the Waymo One Service,” Heliyon (2024).
   Published study.

Back to Mathematics and AI