The Pass Nothing Could Have Failed

I.The First Known Thing — the property that passed because nothing asked

formal verification · model checking of hardware designs

A model checker takes a design — a circuit, a protocol, a controller — and a property written in temporal logic, and decides exhaustively whether every possible behaviour of the design satisfies the property. A typical property is conditional: every request is eventually acknowledged. When the checker answers that the property holds, the answer is a proof, not a sample. For a long time that answer was treated as the end of the matter. A property that passed was a property the design met.

what IBM's verification group found when it looked twice

In 1997 Beer, Ben-David, Eisner and Rodeh, working in IBM's formal verification group at Haifa, described a failure that the proof could not show. A conditional property holds trivially in any design in which its condition never occurs. If, through an error in the design or in the model of its environment, no request is ever issued, then every request is eventually acknowledged — and the checker, correctly, says so. They called such a pass vacuous, and they reported that in the first verification runs on a new design something like one property in five passed in this way. The pass was not wrong. It was empty: it carried no information about the behaviour the engineer had written the property to examine, and nothing in the output distinguished it from a pass that did.

and the instrument that followed

The response was to make vacuity a checkable quantity rather than a matter of care. Kupferman and Vardi gave the general form in 1999: a property passes vacuously in a design if some part of it can be replaced by an arbitrary value — the condition by false, the consequence by true — without changing the verdict. If the verdict survives the replacement, that part of the property did no work. A pass that is not vacuous can be backed by an interesting witness: a concrete behaviour of the design in which the condition actually occurs and the consequence actually follows. A companion idea, coverage, turned the test the other way round and mutated the design instead of the property, asking whether a change to some part of the circuit would have been noticed by any property at all. Neither asks whether the property is true. Both ask whether its truth was ever put at risk.

II.The Second Known Thing — the study that found no effect

drug discovery · toxicity and safety pharmacology

Walk to a hall where a candidate drug is being made safe enough to give to a person for the first time. Before a first human dose, it passes through a package of nonclinical studies. Safety pharmacology, governed since 2000 by the ICH S7A guideline, asks whether the compound disturbs the functions that keep an organism alive in the short term; its core battery covers the cardiovascular, respiratory and central nervous systems. Repeat-dose toxicology, in two species, asks what damage accumulates over weeks. Most of what these studies report, for most compounds, is absence: no effect on respiration, no change in the behavioural observation battery, no histopathological finding in the kidney at the doses tested.

how the field guards its negative results

The field knows that a negative must be earned, and it has rules for earning it. S7A asks that, where no adverse effect is seen on the parameters studied, the highest dose be one that produces moderate adverse effects in this or in other studies of similar route and duration; that measurements be timed on pharmacokinetic and pharmacodynamic grounds; that the test system be pharmacodynamically responsive; and that positive controls be included, though it allows that in well-characterised in vivo systems they may not be necessary. Toxicokinetic sampling confirms systemic exposure, and safety margins are stated as multiples of the exposure expected in patients. For biological drugs the ICH S6 guideline goes further and requires a relevant species — one in which the drug is pharmacologically active at all.

and the record by which the package is judged

The package is judged by how well its negatives hold in people. Monticello and colleagues, pooling data on 182 molecules through the IQ consortium in 2017, found that the absence of a finding in animals was followed by the absence of the corresponding finding in the first human trials about 86 per cent of the time, against a much weaker 43 per cent for positives predicting positives. That figure is the field's main evidence that its clean results mean something. It is computed across all negatives together, and it inherits the base rate: most compounds do not cause most toxicities, so a negative is usually right even when the study behind it could not have seen the effect it denies.

III.The Bridge

no vacuity paper cites a safety study

The halls share nothing. Model checking belongs to logic and hardware engineering; safety pharmacology to physiology and regulatory science. Where toxicology has borrowed method it has borrowed from statistics — power, likelihood ratios, benchmark doses — and the vacuity literature has spread into software testing and requirements engineering but not, so far as a search of either literature shows, towards the laboratory. Nor is the bridge the familiar counsel to include a positive control, which both halls would recognise and which does not by itself say what makes a pass empty. What the two share is a specific structure: a verdict of no violation returned on a conditional claim, where the condition is not a property of the instrument but of the thing examined, and where nothing in the verdict records whether the condition ever held.

X: a pass is worth only the conditions it actually met

Call it vacuous clearance. A safety conclusion of no effect is a conditional claim: if the compound reaches the tissue at the free concentration it will reach in patients, engages whatever it would engage there, during the window in which the consequence would show, then no harm follows. The study returns a verdict on the whole conditional. Where the condition was never realised — a compound that barely crosses the rat's blood–brain barrier but crosses the human one, an in vitro channel assay whose top concentration was capped by solubility below the free exposure it was meant to exceed, an off-target the test species' orthologue does not bind, an observation made after the effect has come and gone — the study passes, correctly, and says nothing. So the information in a negative is set not by the dose, the group size or the assay's sensitivity but by whether its antecedent was witnessed, and that is checkable by the model checker's device: replace the compound, in the argument, with one known to cause the effect by the mechanism this compound would use, at the exposure actually achieved at the site, and ask whether the study as run would have returned a different verdict. If it would not, the pass is vacuous. Two things follow that are not the ordinary counsel to test at high doses. First, the rule S7A uses — moderate adverse effects in this or another study — certifies that the dose was high, not that the particular antecedent of the particular parameter was met; a dose can be toxic to the liver and still leave the brain unreached. Second, vacuity is a property of each conclusion rather than of the package, so a package has a vacuity rate: the fraction of its negative conclusions for which no interesting witness exists. It can be counted, it can be compared across programmes, and it is not reported anywhere.

IV.The Question That Becomes Possible

neither discipline would ask this alone

The safety hall asks whether its negatives predict human negatives, and answers with a single pooled figure. The verification hall has no animals and no tissues, but it learned, from designs that shipped with properties that had never been tried, that a pass must be accompanied by evidence that it could have failed. Neither asks whether the negatives that later fail in people are distributed at random through the safety package, or concentrated in the ones that were never in a position to fail.

the vacuous-clearance hypothesis

Can each negative conclusion in a nonclinical safety package be classified, before any human data exist, as witnessed or vacuous — according to whether the study demonstrates that the compound's plausible route to that effect was engaged at the site, in that species, at relevant free exposure, within the window of observation — and are the organ-system findings that appear in first-in-human and early clinical trials despite a clean nonclinical record concentrated among the vacuous negatives, to a degree that systemic exposure multiples and assay sensitivity do not already explain?

why it would matter

Because the pooled figure is doing work it may not be able to bear. An 86 per cent negative predictive value can be produced by a package whose witnessed negatives are nearly always right and whose vacuous negatives are right only at the base rate; the first kind is evidence and the second is luck, and the pooled number cannot tell them apart. If the hypothesis holds, the field has a way to know, study by study and before dosing anyone, which of its clean results are clean because they were tested and which are clean because nothing was asked — and the vacuous ones are exactly where extra monitoring, a different species or an in vitro system with the human orthologue would buy the most. It also makes the argument about replacing animal studies more precise. A new approach method need not match an animal study's verdicts; it needs to witness the antecedents that the animal study left unmet, and that is a specification that can be written down.

V.How It Could Be Proven — or Broken

a candidate must be breakable, or it is merely poetry

An intercognix candidate earns nothing until it names its own test. This one needs no new study. It needs nonclinical packages whose individual conclusions can be read in detail, paired with the adverse events observed in the first human trials — the kind of pairing the IQ consortium and the eTOX and eTRANSAFE projects have already assembled — and the patience to read each negative for its antecedent.

the honest status

Note the narrowness. The claim concerns one kind of statement — a nonclinical conclusion of no effect on a named organ system — and one downstream event, a clinical finding in that system in early trials, for which ground truth exists and has already been pooled at least once. It does not claim that animal studies predict badly, and it does not claim that any particular study was inadequate; S7A's own rules, and S6's insistence on a relevant species, show that the field already enforces non-vacuity in the places where it has noticed the problem. It claims only that the same check, made per conclusion and made explicit, sorts negatives into two populations with different reliabilities, and that the pooled figure has been averaging them. It is also worth saying what does not transfer. A model checker's verdict is exhaustive over the design and its replacement test is exact; a toxicology study samples a few animals at a few times, and its conditions can only be argued, not computed. What transfers is a discipline of reading — that a clean verdict on a conditional claim must be accompanied by evidence that the condition occurred — and not the proof that makes the discipline clean in silicon. The whole thing rests on a comparison that a pooled nonclinical-to-clinical database would allow: whether the negatives that failed in people were disproportionately the ones that could not have failed in animals. If they were not, this is a relabelling of the safety margin, and should be said so plainly. Until that comparison is made, this remains exactly what its stamp says: a candidate. A structural echo between two halls of the library, waiting for someone with the tools of both to walk the corridor.