How p-hacking built the world's most expensive safety regime

The case against nuclear power rests on a handful of studies that claim that low-dose-rate radiation causes cancer. There is just one problem: that’s not what the studies actually show.

Issue 24 of Works in Progress has now arrived with subscribers. Sign up here to receive this issue, plus another every two months – straight to your door.

---

Chernobyl was the world’s worst ever nuclear disaster. The reactor was built to a flawed design. In 1986, its operators disabled automated safety systems and removed control rods to perform a late-night test. The coolant water overheated, speeding up the reaction until it flash-boiled into steam and violently shattered all the nuclear fuel in a matter of seconds. The graphite in the reactor then caught fire.

Six hundred workers were present when this occurred. Of these, 134 received high doses of radiation, between 800 and 16,000 millisieverts, mostly within hours. Twenty-eight of these died in the first three months, and another 19 died before 2004, though not all from radiation-related causes. While first responders have shown a slight increase in rates of leukemia, there has been no increase in solid cancers.

When the reactor exploded, it released massive amounts of radioactive material into the atmosphere. This ash settled onto the grazing regions of Ukraine. Dairy cows ate it and secreted it into their milk. Every person who drank this milk ended up with some quantity of radioactive iodine concentrated in their thyroid gland. Since children have smaller thyroids, their effective doses were much larger. The hundreds of thousands of children exposed have been closely screened in the years since, and authorities have identified 6,000 thyroid cancers. Fifteen people have already died of these cancers, and 200 might in the long run. These thyroid cancers were completely avoidable because the iodine has a half-life of eight days. After the Windscale fire released radioactive fallout onto grazing land in 1957, Britain discarded contaminated milk for 44 days, thereby avoiding any harm.

Chernobyl is the only accident in commercial nuclear history that has exposed people to large enough doses of radiation to poison and kill them. But even it has caused only hundreds of early deaths, despite the exposure of millions of people in the exclusion zone and nearby. Radiation impacts on Scandinavia and Germany, where there were major fears about the effects of the fallout, were nugatory. Evacuations and relocations to avoid small additional background radiation levels may have caused more harm than they averted. The same is true of Fukushima and Three Mile Island, the other two large nuclear disasters, but to an even greater extent: neither saw any responders die of the direct effects of radiation, and neither shows any clear impact on cancer rates.

Two years before Chernobyl, an explosion at a pesticide plant in Bhopal, India, released toxic methyl isocyanate gas that killed at least 2,000 people instantly, permanently disabled another 4,000, and caused 550,000 injuries in total. In 1975, the Banqiao Dam in China failed, flooding 12,000 square kilometers, drowning at least 25,000 people, and destroying perhaps five million houses.

Whereas Chernobyl is a household name, Bhopal and Banqiao are mostly familiar only to specialists. People have much greater familiarity with and concern about the risks created by nuclear power, and the world’s international radiation protection regime is based on the idea that any release of radioactive material from a nuclear power plant is intolerable. This has led to regulations that have increased the costs of nuclear electricity over time to the point where it is widely considered a slow, backward, and ineffective technology.

The idea that any release of radioactive material is an intolerable disaster rests on the claim that radiation is harmful even in small, spread-out doses. But this claim is not well supported. The studies that claim to show harms from slow, drawn-out ionizing radiation are unconvincing. By giving them undeserved credence, we may be foreclosing one of the world’s most powerful technologies.

Trouble in Taipei

Between 1982 and 1984, recycled steel contaminated with cobalt-60 was unknowingly used in the construction of more than 180 buildings in Taiwan, including schools, businesses, and approximately 2,000 apartments. Cobalt-60 emits deeply penetrating gamma radiation, which in high doses is fatal: in 2000, three people died from acute radiation poisoning when a Thai scrap merchant took apart a radiotherapy machine containing large amounts of it. Over two decades, more than 10,000 people occupied these buildings, receiving an average total radiation dose of 400 millisieverts, about seven times more than the average American would receive over the same period from background radiation. This makes the Taiwanese buildings a good test of the claim that chronic background radiation at low daily rates damages human health.

A series of academic papers reconstructed the doses that different affected inhabitants experienced, combining thousands of on-site readings with estimates of their time spent at home. CJ Tung and colleagues built a model in 1998 of 24 of the 1,327 apartments that had then been identified as irradiated, breaking them down into one meter by one meter squares, and using dosimeters to measure radiation in those squares, to get an idea of how residents might have been affected.

Doses varied substantially. Most units received trivially low doses, less than the radiation exposure from one full-body medical CT scan per year (ten millisieverts). A small fraction of units received nearly 1,000 millisieverts per year, equivalent to a hundred CT scans. Later studies that mapped out these doses in more detail found that some sections of rooms were exposed to over 8,000 millisieverts per year, equivalent to about two CT scans every day.

A 2004 preliminary report, followed by a 2006 study (Chen et al), was the first to attempt to measure the health effects in this group. There were no cases of radiation sickness nor any major chromosomal aberrations. Affected Taiwanese had biomarkers of cellular stress nearer the upper end of normal reference ranges, but none linked with observable health effects. Cancer rates were, unexpectedly, dramatically lower than in the population at large.

Chen interpreted this as evidence of the health benefits of radiation. This theory, known as hormesis, holds that low doses of stressors, including ionizing radiation, can improve health (in this case, reducing cancer risk) by triggering the body’s repair systems in much the same way that exercise improves fitness by stressing the cardiovascular system. While popular among a small community of researchers, it has not gained widespread acceptance due to limited and conflicting evidence in humans.

This interpretation of the Taiwan data is dubious because it wasn’t able to control for age. Those affected by the cobalt-60 radiation were far younger than the average Taiwanese. Since younger people are less commonly diagnosed with cancer, this is a very significant limitation. A US Environmental Protection Agency (EPA) review, describing the Taiwanese apartment data alongside some similar cases, warned that ‘studies on these populations are ongoing and suffer from various shortcomings, including incomplete follow-up, dosimetric uncertainties, limited statistical power and confounding’.

Torture the data and it will confess

Later studies fixed most of these problems, linking good estimates of the doses individuals experienced with their health outcomes and adding the missing age data. Contrary to the previous report, the studies generally claimed to find that low doses of ionizing radiation can cause cancer. But drawing this conclusion from them requires an unreasonable manipulation of statistics.

A 2006 study (Hwang et al) in the International Journal of Radiation Biology, for example, found slightly elevated incidences of thyroid cancer in women and leukemia in men across a sample of 7,271 people. But overall cancer rates were still lower in this group than in the general Taiwanese population, even after controlling for age.

The way the researchers found these higher rates of cancer was to break cancer cases down into 77 subtypes, such as tongue cancer in men or kidney cancer in women, and see if any of these had higher rates than expected. This approach has two problems. The first is that when you divide cancer cases into 77 buckets, even small and random variations in one or two of the buckets can create the appearance of a relationship. Every statistically significant site-specific result showing higher cancer incidence relies on seven or fewer observed cases. Across 77 different statistical tests, we would expect to see statistically significant results in four cases purely due to random variation. The second problem is that the authors did not register their hypotheses in advance: we have no idea whether they sliced the data into hundreds of different tests before cherrypicking those that gave them statistically significant correlations.

The biggest issue with drawing the inference that low-level radiation harms human health is that the exposed population actually have lower cancer rates than unexposed people. Just like in Chen et al, the number of total cancer cases observed is substantially lower than expected.

Don’t back down, double down

A 2017 study in the British Journal of Cancer (Hsieh et al) follows up on this result, with ten years more data on whether affected people got ill, again using more accurate dose-rate profiles, more closely associated with particular individuals. Like Hwang et al, they claim their findings show that ‘low-dose-rate radiation exposure increases risks of breast cancer and leukaemia’, but the data is similarly brittle.

Within their sample of around 6,000 exposed Taiwanese apartment dwellers and schoolchildren, female breast cancers were higher at higher levels of exposure: 100 millisieverts of exposure was linked to an 11 percent higher cancer risk. This result is statistically significant. The study finds a few other results that are narrowly significant, but only out of a table of dozens, just like the previous study.

But again, the studies fail to state which tests they planned to do beforehand or that they began the analysis with the hypothesis that leukemia (but excluding multiple myeloma and chronic lymphocytic leukemia) would go up after a delay of two years, whereas rectal, stomach, liver, and thyroid cancer would be unaffected – or that cancers overall would be no higher in the study group than among totally unaffected Taiwanese. If they’d entered their research with this hypothesis, it might make the results more credible. Instead, they appear to have stumbled across a handful of correlations by chance.

And once more, the authors don’t highlight that the irradiated group has significantly lower overall cancer rates than the general Taiwanese population.

While we can comb through the data and infer that the average age of the subjects in Hwang was 33. The samples are slightly different, so we can’t just add ten years to get an average age of 43 for Hsieh. The UN Scientific Committee on the Effects of Atomic Radiation estimated that the average age of the Hsieh sample was about 45, seven years older than Taiwan’s population overall in 2012. By 2012, the irradiated apartment dwellers had experienced 247 cancers, across 97,106 person years. Yet the cancer rate for Taiwan as a whole between 2008 and 2012 was about 390 per 100,000 person years.

Overall, cancer rates seem to be almost 35 percent lower in the affected group. If being irradiated with hundreds of millisieverts per year appears to have no effect, or even reduces cancer rates compared with the general population, this strongly implies that the links these two papers identify between much smaller doses of radiation and particular kinds of cancer are random noise.

Made aware of this, Hsieh responded (as did Hwang) that these gaps must be down to socioeconomic status. Is this true? The best argument provided is that these were residents of new buildings constructed during an economic boom, suggesting that the apartments were relatively expensive. Though no data on the actual socioeconomic status of their residents exists, we can make some educated guesses.

First, though new, some of the buildings were public housing intended for poorer residents or public employees. This makes it unlikely that their inhabitants were wealthier people projected to have better health outcomes than the rest of the population.

Second, even if it were true that the residents were richer than average, this still wouldn’t account for the decreased cancer rates. Existing data suggest that Taiwanese health differences by socioeconomic status do not close this gap: overall cancer incidence is ten percent lower among doctors than in the general population. The incidence of colorectal cancer, one of the most common cancers, is actually lower (by about 20 percent) at the bottom of the wage ladder in Taiwan, though it is higher among those with the lowest education. None of these variations approach the 35 percent lower rates of cancer seen in the irradiated apartment residents.

This is not to claim that Hsieh’s paper provides strong evidence for the health benefits of radiation, as some fans of nuclear power have suggested. Most likely, these results are evidence of nothing.

The perils of low-dose science

From 2011 onwards, scientists began to realize that significant numbers of famous, eye-catching findings were unreplicable, in what became known as the replication crisis. Starting in psychology – once-popular ideas like power posing or the brain-enhancing effects of classical music began to be debunked – before the same occurred across every discipline.

The Taiwan papers have the classic tell of such findings, using tricks to turn random noise into statistically significant results, a phenomenon known as p-hacking. One such trick is dicing a population up into as many different subgroups as possible, expanding a researcher’s likelihood of finding something by chance.

It is inherently difficult to find out if low doses of radiation affect health. Populations vary hugely in all sorts of health, lifestyle, and genetic factors. The anticipated cancer effects of radiation would be low, even for those who expect to find them. Researchers would need a huge sample size and very carefully designed analysis to have any hope of detecting a signal amid the noise. INWORKS, first published in 2015 and updated in 2023, is the most serious attempt. It looks at cumulative low-dose radiation among 300,000 nuclear workers in France, the US, and Britain, measured by badge dosimeters, and finds that each additional 100 millisieverts of lifetime ionizing radiation exposure is associated with five percent higher cancer mortality rates, adding about one percent to the lifetime risk of dying of cancer (compared to an average of about 34 percent).

For comparison, 100 millisieverts is about ten full-body CT scans, whereas the median INWORKS worker experienced a four-millisievert dose. Eating 50 grams of processed meat per day is associated, after controls, with a lifetime colorectal cancer risk 18 percent higher, and smoking with a lung cancer risk 1,500 to 3,000 percent higher.

The INWORKS study is the best single piece of evidence for an effect of cumulative low-dose-rate radiation on health, but the size of the effect is small enough that it is far from a given. For one, the INWORKS study does not attempt to account for background doses, despite the fact that different people will have different levels of background exposure, depending on factors like where they live and how many medical scans they have. The average lifetime background dose in the UK is about 70 millisieverts, similar to that in the US and France, meaning that more than 90 percent of INWORKS subjects got most of their exposure from natural background radiation.

Another problem is that INWORKS’s significant result is driven by subjects between the 75th percentile, meaning lifetime exposure of roughly 20 millisieverts, and the 90th, at roughly 50 millisieverts. The relationship between exposure and cancer is not significant for doses between 0 and 20 millisieverts. Only when expanded to doses between 0 and 50 millisieverts does a significant effect appear, before weakening steadily as higher doses are added in, though not to the point of becoming insignificant.

This sensitivity makes the results fragile and is one reason why many look instead at studies with higher doses. The median INWORKS worker received a dose of four millisieverts, which according to INWORKS’s model, would correspond to increasing the lifetime risk of dying from cancer from the US average rate of 17.2 percent to 17.23 percent. An effect this small is hard to confirm even with 300,000 subjects, given so many unmeasured confounders.

Historically, many people have received doses of thousands or even tens of thousands of millisieverts, similar to those of the Taiwanese apartment dwellers. There are cases of Keralans exposed to sand containing thorium; Manhattan Project scientists who inhaled so much plutonium that they peed it out; early British radiologists who worked without any protection from the X-rays they worked with. These groups suffered long-term health effects only if they absorbed high dose rates.

One of the most horrific and notorious cases of radiation exposure was that of women employed to paint radioactive radium onto watches for glow-in-the-dark dials. Early dial painters were told to lick their paintbrushes for greater precision. Those that did ingested radium at extremely high internal doses and developed necrosis of the jaw and often-lethal bone cancers. But the thousands that did not lick their brushes, despite receiving doses that were often hundreds of times higher than normal, did not see any higher rate of cancer.

Though unlikely, it is possible that future studies will reveal a more substantial correlation. In the meantime, the lack of any strong signal in the Taiwan data is one of many good indicators that low-level radiation is not a cause for concern. Where radiation does harm people is when they are exposed to high, acute levels that even the worst nuclear disasters seldom release. When people are looking this hard, absence of evidence is, in fact, pretty good evidence of absence.

---

Alex Chalmers is a staff writer at the Cosmos Institute and a former editor at Works in Progress.

Ben Southwood is an editor at Works in Progress.