The Count the Catalogue Cannot Give

I.The First Known Thing — the set whose size is not the number of things in it

psychology of identification evidence · nominal, functional and effective size

An identification procedure presents a witness with a set: one person whose identity is at issue and a number of others included only to furnish alternatives. The protective arithmetic is supposed to be obvious. Put six people in the set and a witness with no memory at all has one chance in six of picking the person at issue, so the procedure converts a certainty into a probability, and the larger the set the stronger that protection. This is the reasoning that produced rules of thumb about how many members a set should contain, and it rests on counting them.

the counting objection

The counting is wrong, and the field established why in the late nineteen-seventies. The alternatives only work as alternatives if they are alternatives: if the others plainly do not answer the description the witness gave, the set has six members and one choice. The number of members is called the nominal size. The number that functions is not a property of the set that any catalogue of its contents can report, because it depends on the description under which the choice is being made. A set can be enlarged indefinitely without becoming any more protective, and a set of six can be worth less than a set of two.

the measurement, and the odd thing about who performs it

What makes this a usable quantity rather than a complaint is the procedure the field devised to measure it. Participants who witnessed nothing are given the verbal description and nothing else and asked to choose. These are the mock witnesses: choosers equipped with the selecting description and denied all access to the answer. Their choices fall where the description permits them to fall, so the distribution of their choices over the members of the set reads off the number of members the description admits. Wells, Leippe and Ostrom, in 1979, defined functional size as the reciprocal of the proportion of these uninformed choosers who land on the person at issue: if a third of them land there the set is functioning as three, whatever its nominal size. Malpass, in 1981, proposed effective size, which reduces the nominal count by the extent to which members are chosen below chance, so that members nobody would ever select are discounted rather than counted. Tredoux, in 1998, supplied inferential statistics for these measures, including a dispersion index for which confidence intervals can be constructed and independent sets compared.

what the measure is and is not

Two features of this apparatus matter later and are easy to lose. The first is that it is a measurement of the set, taken through the readout that will be used on it, by an agent who cannot be right. Its validity does not depend on the mock witnesses being perceptive, careful, or representative of real witnesses; it depends only on their being uninformed. The second is that the quantity is not a correction applied to a verdict. It is a scale against which a selection is read: the same choice, made from sets of the same nominal size, carries different evidential weight according to how many alternatives the description admitted, and that weight can be driven to nothing by composition alone without anyone behaving improperly or any instrument performing badly.

II.The Second Known Thing — the hit quoted against a million

drug discovery · hit finding, screening and library design

A screening campaign presents an assay with a set too: a deck of compounds, often several hundred thousand and sometimes more than a million, each tested once at a single concentration, with the readout a number per well. Compounds whose number crosses a threshold — conventionally three standard deviations from the distribution of the inactive population, with assay quality certified beforehand by the Z-prime factor of Zhang, Chung and Oldenburg, published in 1999 — are called primary hits. Hit rates are quoted in tenths of a per cent, and the nominal size of the deck is quoted alongside them as the measure of how much chemical space the campaign interrogated.

what the field already knows about its own deck

The field is not naive about the deck, and the practices that follow must be stated at strength because the candidate has to survive them. It has known for two decades that a sizeable share of primary hits arise from mechanisms that have nothing to do with the target: colloidal aggregation, which inhibits unrelated enzymes and is diagnosed by detergent sensitivity, reported from Shoichet’s laboratory; reactivity, redox cycling, and interference with the detection chemistry itself. Baell and Holloway, in 2010, derived four hundred and eighty substructure filters from the behaviour of some ninety-three thousand compounds in one detection technology, and gave the resulting class its name. Compounds that appear in many unrelated campaigns are tracked as frequent hitters. Counter-screens are run against the detection technology without the target, and against unrelated targets, specifically to find and remove such compounds. Primary hits are confirmed by retest, by dose–response, by an orthogonal readout, and increasingly by a biophysical method that reports binding directly.

how the output of those controls is used

Note carefully what is done with those results, because this is the whole of the opening. Every one of those controls is used subtractively. A compound that lights up the target-free run is struck off; a substructure on the filter list is flagged; a frequent hitter is deprioritised. The output is a blacklist, and the blacklist is applied to individual compounds. The distribution of the target-free run across the deck — which regions of it the readout is willing to select when there is nothing there to be found — is discarded once the offending wells have been identified. No number is carried forward from it, and nothing about the surviving hits is rescaled by it.

the nearest thing the field has, and why it is not this

One practice comes close and must be set aside explicitly. Benchmarking sets for computational screening, of which the directory of useful decoys is the best known, deliberately surround each known active with compounds chosen to match its physical properties while being presumed inactive, precisely so that a method cannot score well by sorting on size or greasiness. That is the same intuition about alternatives, and it is the closest neighbour in the literature. But the decoys are constructed by property matching rather than measured by an uninformed run; the quantity produced is a figure of merit for an algorithm, not a weight attached to a particular experimental hit; and the construction has been shown to introduce its own bias, which is a problem one does not have when the alternatives are read off the readout’s own behaviour. The deck itself, meanwhile, is described by scaffold counts and diversity indices, which are computed from structure alone and therefore cannot know what any readout admits.

III.The Bridge

no paper on line-up fairness cites a paper on compound screening; both halls present a chooser with a set and quote the result against the number of members, and only one of them has stopped doing so

The halls do not meet. The measurement of set fairness is forensic psychology and legal procedure; primary screening is pharmacology and assay technology. Neither cites the other and no review spans them. Three further neighbours have to be set aside. The hit rate is a ratio of selections to members and is computed from the very run whose weight is in question, so it counts choices and never alternatives. The Z-prime factor measures how cleanly the assay separates its two control populations, which is a statement about the instrument and contains no information about the composition of the deck. Filters and frequent-hitter lists do concern the deck, but they are properties attached to compounds and summed into a count of bad actors, and a count of the members to be removed is not a count of the members that remained available.

X: a selection is worth the alternatives it declined, and that number is measured by a chooser who is given the description and denied the answer

Call it the admissible alternative count. Where something is selected from a presented set by a readout, the evidential weight of the selection is carried by the number of members the readout would have been willing to select, and that number is a property of the readout and the set jointly rather than of either alone; so it cannot be obtained from any catalogue of the set’s contents, however complete, and it is not raised by enlarging the set with members the readout would never have taken. Three things follow, and they are not one thing said three times. The first is that the quantity is nevertheless measurable, by exactly one procedure: run the same readout over the same set with the answer removed, and read the admissible count off the dispersion of what it then selects. The informed run reports a selection; the uninformed run reports the scale that selection should be read against. The second is a reversal of use. Where a field already performs the uninformed run, it will have been using it to strike members off, and striking members off throws away the dispersion, which is the part that carries the scale; the same experiment, already paid for, yields a weight for every surviving selection instead of a verdict on the discarded ones. The third is that the weight is local. The dispersion of the uninformed run is not uniform over the set, so a selection falling where that run concentrates was taken against few alternatives and one falling where it is sparse was taken against many, and two selections of identical magnitude from the same set under the same readout can therefore differ in weight by an order the magnitude does not show. What the pairing makes available is not a cleaner blacklist but a denominator: the number a result should have been quoted against all along, in place of the number of things that were present.

IV.The Question That Becomes Possible

neither discipline would ask this alone

The psychologist has no reason to ask it. The sets are small, the readout is a human judgment, there is no magnitude attached to a choice, and nobody has a million members to characterise. The screener has no reason either: the target-free run is housekeeping whose purpose is to remove compounds, a hit’s credibility is already expressed as a potency and a confirmation status, and the deck’s size is a selling point rather than a denominator anyone suspects. Held together, one question becomes available, and it is answerable from data that large screening operations have already generated and stored.

the admissible-count hypothesis

For a fixed deck and a fixed detection technology, compute from the target-free run a single dispersion statistic over structural classes — the reciprocal of the sum of squared selection proportions, the same form the identification literature uses — and call it the deck’s admissible alternative count for that readout. The hypothesis is that this count predicts what fraction of primary hits survive orthogonal confirmation, across campaigns run on the same deck with the same technology against different targets; that it does so after the hit rate, the Z-prime factor and the deck’s scaffold diversity are held fixed; that the prediction also holds within a campaign, so that hits falling in structural neighbourhoods where the target-free run concentrates confirm at materially lower rates than hits of equal potency falling where it is sparse; and that the count is a property of the pairing rather than of either side, changing when the technology is changed at fixed deck and when the deck is changed at fixed technology.

why it would matter

Because the decision a screening operation actually faces is where to spend confirmation effort, and it currently allocates that effort by potency and by blacklist membership, both of which are properties of single compounds. A local admissible count would reorder the queue: it says that two equally potent hits can deserve very different amounts of scepticism because of what the readout could have chosen around them, and it says so from an experiment the operation has already run. It also puts a number on something decks are sold by. If enlarging a deck with members a given technology will not select leaves the admissible count unmoved, then the acquisition raised the nominal size and bought no evidential weight for that technology, and the operation could have known which acquisitions those were before making them.

V.How It Could Be Proven — or Broken

a candidate must be breakable, or it is merely poetry

An intercognix candidate earns nothing until it names its own test. This one needs no new chemistry, no new assay and no new instrument, only archived plate-level data of a kind that screening centres retain: target-free or unrelated-target runs of a detection technology over a deck, the primary hit lists of campaigns on that same deck and technology, and the recorded outcome of confirmation for each hit.

the honest status

Note the narrowness. The claim concerns the weight of a primary selection under one readout over one deck, and it is silent on everything a programme cares about afterwards: whether a confirmed hit is tractable, selective, or worth a chemist’s year. It does not say that low-count hits are false, only that they were taken against fewer alternatives than the deck’s size implies and should be read against that smaller number. Nor is the measurement available everywhere, since it requires a readout that can be run over the whole set with the answer removed, which rules out campaigns whose detection chemistry cannot be separated from the target. What transfers is narrower than either hall’s ambitions. A selection is worth the alternatives it declined. That number is not in the catalogue of what was present, and the only instrument that reports it is a run of the same readout by a chooser who could not have been right.