The Failure That Only Came Sooner

I.The First Known Thing — the birth rate that fell while no one had fewer children

demography · the quantum and tempo of fertility

The total fertility rate is the figure most often quoted about a population’s childbearing, and it describes no woman who has ever lived. It is made by taking the birth rates observed in a single year at each age of the reproductive span and adding them, which gives the number of children a woman would bear if she passed her whole life at that one year’s rates. Demographers call the construction a synthetic cohort. Its merit is that it is available at once, where the completed family size of real women born in a given year cannot be known until they are nearly fifty. Its defect was set out by Norman Ryder in the 1960s under the name of demographic translation: a period measure and the cohort quantity it stands in for part company whenever the timing of the event is on the move.

the factor, and what it does and does not contain

In 1998 John Bongaarts and Griffith Feeney gave the departure a form simple enough to use. They separated the quantum of fertility, how many children are born, from its tempo, when in life they are born, and showed that if the mean age at childbearing is rising by r years in each calendar year, the births that would have fallen within one year are spread over rather more than one, so that the year receives only the fraction 1 − r of them. The observed rate is the undistorted rate multiplied by that fraction, and dividing by it, birth order by birth order, removes the distortion. The factor contains the rate at which the timing is changing and nothing else. It does not contain the timing. A population whose mothers are old but no longer getting older shows no distortion at all; one whose mothers are getting older by three months a year shows a rate a quarter below its undistorted value, whatever their age.

what happened when it was applied

The correction changed the reading of a continent. Through the 1990s period fertility across much of Europe fell to levels without precedent, and by 2003 twenty-one countries stood below 1.3 children per woman, a condition named lowest-low fertility and widely taken for a collapse in the wish to have children. In the Czech Republic the rate reached 1.13 in 1999. The tempo-adjusted figure for the same country in 2004–2006 was 1.78: most of the fall had been first births moved to later ages and not births forgone. When the rise in the age at childbearing slowed, the period rate recovered with it, to 1.44 in the Czech case by 2007, and Goldstein, Sobotka and Jasilioniene recorded in 2009 that a single European country remained below the line. The mothers had not become younger again. Their age had only stopped rising so fast.

what a synthetic cohort can be made to say, and the limits the field set on its own correction

Run the other way, the same construction produces impossibilities. Where an event is being brought forward, a period sum gathers the late events of older cohorts and the early events of younger ones into the same year, and the textbook illustration is a synthetic cohort of whom a hundred and twenty per cent marry. The correction has been contested within demography, and its limits are part of what is known. It assumes that the schedule moves bodily without changing shape; Kohler and Philipov showed in 2001 that a change in the spread of ages biases it. Comments published in 2000 argued that the rates should be taken on those actually at risk and not on all women, and questioned how closely the adjusted figure follows what cohorts go on to complete. And its authors were explicit that the adjusted figure is a period measure with one distortion removed, not a forecast for any real cohort.

II.The Second Known Thing — the success rate read off a single decade

drug discovery · portfolio direction, programme termination, and herding

The figure most often quoted about drug development is the likelihood of approval, and it is built in the same way. Hay and colleagues, in 2014, took 5,820 phase transitions recorded between 2003 and 2011, defined the success of a phase as the number of programmes that advanced out of it divided by the number that advanced plus the number suspended, and multiplied the phases together. The industry’s standard report for 2011–2020 did the same over 12,728 transitions in 9,704 programmes: 52.0 per cent out of Phase I, 28.9 out of Phase II, 57.8 out of Phase III, 90.6 at filing, and a likelihood of approval from Phase I of 7.9 per cent. A later analysis of phase-transition data for 2014–2023 gives 6.7 per cent. These are the figures a portfolio committee multiplies into every valuation, the figures by which a therapeutic area or a modality is judged to be getting harder, and the figures behind the statement that success rates are falling.

what kind of number it is

No programme lived through those rates. The same report puts the average passage from Phase I to approval at ten and a half years, which is the length of the window: the Phase I transitions counted in the decade belong largely to programmes begun inside it, and the Phase III and filing transitions largely to programmes that entered the clinic before it opened. The product is the fate of a hypothetical programme exposed at each phase to the events of one period, which is a synthetic cohort in the demographer’s exact sense, with phases where the fertility table has ages. And each phase figure is not a rate on programmes at risk. It is a ratio of two counts of events that happened to fall inside the window, advances and suspensions, each arriving by its own clock.

the clocks are not still, and the field has worked to move them

Those clocks are objects of policy. The account of research productivity published by Paul and colleagues in 2010 set out a paradigm of quick win, fast fail, in which technical uncertainty is resolved earlier and attrition is shifted towards the cheaper phases. The analysis that reports the 6.7 per cent attributes part of the decline to the use of biomarkers, which allow programmes not showing the desired biological effect to be terminated early, and commends designs that encourage attrition at the earliest stages; it reports that success in Phase I has gone from over 75 per cent in 2006–08 to below 40. The decline is read in one way, as more programmes failing. That the same programmes might be failing sooner, and that a ratio of counts taken in a window cannot tell the two apart, is not among the readings offered.

what the field does know about timing

Timing is known to these analyses as a nuisance. Hay and colleagues noted that the dating of suspensions was not precise enough to analyse yearly changes. Wong, Siah and Lo, who rebuilt development paths from trial records for 2000–2015 and declared a programme terminated once a fixed interval had passed without a further trial, warned that their final year overstated success because too little time had elapsed for failures to be recognised; their rolling windows show overall success falling from 11.2 per cent in 2005 to 5.2 in 2013 and then turning upwards. The Tufts estimates avoid the window altogether by following cohorts defined by entry into the clinic and treating the unfinished as censored, at the price demography also pays, that a cohort’s figure is to be had only when the cohort is old. What none of them supplies is the pair of series a demographer would ask for first: the mean time from entering a phase to leaving it by advance, and the mean time to leaving it by termination, year by year.

III.The Bridge

no paper on the quantum and tempo of fertility cites a paper on clinical success rates; both halls read a whole-course quantity off the events of one period, and only one of them has worked out what moving the timing does to it

The halls do not meet. The tempo literature belongs to population studies and is argued over birth orders, postponement and the meaning of replacement; the success-rate literature belongs to pharmaceutical economics and is argued over attrition, productivity and the cost of a new medicine. Neither cites the other and no review spans them. Three neighbours must be set aside before the proximity is worth anything. Censoring is not the bridge: the unfinished programme at the end of a window is a boundary problem, known and corrected for, whereas the distortion here arises with every event fully ascertained and in the middle of the window. The distinction between period and cohort estimates is not the bridge either; the field has both kinds, uses one for convenience and the other for rigour, and has not stated the sign or the size of the difference between them. And the difficulty of telling a slow programme from a dead one concerns a single programme and the length of a wait. The bridge is an aggregate law, about what a moving clock does to a count.

X: where a whole-course quantity is read off the events of one period, a drift in their timing changes the count falling in the period by a factor fixed by the speed of the drift, whatever number of events any cohort will have; and where the quantity is a ratio of two kinds of exit, its error is the ratio of their two factors

Call it tempo distortion. Where a quantity belonging to a whole course — children per woman, approvals per programme — is assembled from the events of a single period, a drift in the timing of those events alters the number falling within the period by the factor one minus the yearly change in their mean timing, and it does so without altering how many events any real cohort will have. The factor depends on how fast the timing is moving and not on where it stands, so the distortion lasts exactly as long as the movement and vanishes when the movement stops, at whatever timing it stops. Where the period quantity is a ratio between two competing exits from a stage, each exit carries its own factor, and the period odds of the one against the other are the cohort odds multiplied by the ratio of the two factors: a drift common to both exits cancels, and only the difference between their drifts is seen. And where stages are multiplied into a whole-course figure, the distortions multiply with them.

three consequences, not one said three times

Three things follow. The first is a sign and a ceiling. An industry that brings its terminations forward while its advances keep their pace will show a falling success rate in every window for as long as the bringing-forward continues, with no additional failure anywhere; one that lets its terminations drift later will show a rising one. The arithmetic is not small. If the mean time to termination shortened by a tenth of a year in each year, about five weeks, at each of the three clinical phases, phase rates truly at 52.0, 28.9 and 57.8 per cent would be recorded as 49.6, 27.0 and 55.5, and a likelihood of approval of 7.9 per cent as 6.7. The ceiling is as definite: for Phase I success to fall from 75 per cent to 40 by tempo alone, terminations would have to arrive three and a half years sooner with every year that passed, which no phase is long enough to permit. Tempo can account for a bounded part of a fall and no more. The second is a recovery that nobody will have earned. A termination cannot be brought forward indefinitely, since it cannot precede the experiment that justifies it; when the shortening stops the period rate returns to the cohort value with no change in the science, as European fertility recovered when postponement slowed, and it will be read as an improvement. The third is that the comparisons on which capital moves are comparisons of period ratios. A therapeutic area, a modality or a target class in which earlier readouts have lately shortened the time to a termination will show the steepest fall in measured success, and sponsors who steer by the benchmark will leave it; the areas so marked are the ones that have learnt to fail cheaply, which is the conduct the benchmark was meant to encourage.

IV.The Question That Becomes Possible

neither discipline would ask this alone

The demographer has no reason to ask it. His event has one kind of occurrence and his distortion acts on a count; a ratio between two competing exits, each with a drift of its own, is not a fertility problem, and he has no interest in what a company should do with a compound. The analyst of development risk has no reason either, because his variable is the rate. Time in phase is tabulated in the same reports, as a cost, on another page; it is not differentiated with respect to the calendar and it is not divided into the rate. Held together, the halls pose a question that is neither’s: how much of the recorded movement in clinical success is a change in how many programmes fail, and how much is a change in when their failure is declared.

the tempo-adjusted success hypothesis

Concretely. Take a programme-level record of clinical development with a dated entry into each phase and a dated exit from it, by advance or by termination, over two decades. For each phase and each calendar year compute the mean time in phase of the programmes that advanced in that year and of those terminated in that year, and take the yearly change of each, centred on the year, as the two drifts. The claim has three parts. First, that the two drifts differ measurably from zero and from each other over sustained runs of years. Second, that for periods old enough for their programmes to have since finished, the departure of the period ratio from the ratio realised by the matching entry cohorts has the sign the drifts predict and roughly the size, so that the period odds divided by the ratio of the two tempo factors lie closer to the cohort odds than the raw period odds do. Third, that across therapeutic areas the fall in period success is steeper where the termination clock has shortened faster relative to the advance clock, and that where the shortening has stopped the period rate has turned upwards without any change in the cohort rate. The second part tests the arithmetic. The third is what separates the claim from a general remark that timing matters.

why it would matter

Because the benchmark is used to decide. If the distortion is of the size the arithmetic allows, a sponsor that adopts early stopping rules is marked down by the industry’s own measure for adopting them, a class is judged to be failing more when it is failing faster, and a recovery in the benchmark will be credited to whatever was introduced in the years before it. The correction asks for nothing that is not already collected: two columns of durations, differenced by year. It would also give the cohort figure, which arrives a decade late, a period companion that can be had at once and whose error is stated.

V.How It Could Be Proven — or Broken

a candidate must be breakable, or it is merely poetry

An intercognix candidate earns nothing until it names its own test. This one needs no experiment. It needs dated programme histories, which exist in the commercial pipeline databases from which the benchmarks are made and can be rebuilt from public trial registries in the manner of Wong and colleagues, and it needs enough elapsed time for the earlier cohorts to have finished, which they now have.

the honest status

Note the narrowness. The claim concerns a ratio of event counts taken within a window, and the product of such ratios. It does not say that drug development has not become harder; the ceiling computed above means that the larger reported declines cannot be tempo, and the claim is confined to the part that can. It does not assert that terminations have in fact been brought forward, which is the first thing the test measures and may prove false. That a drift of five weeks a year reproduces the difference between two published figures is an illustration of size and carries no evidential weight. The claim says nothing against estimates built on entry cohorts, which are free of the distortion and pay for that in delay. And where the two clocks are steady the factor is one, the adjustment changes nothing, and the proposal has nothing to say, which is how it ought to behave.