Research methods · The literature

Publication bias, funding & conflicts of interest

You're trying to decide whether a supplement does anything. So you do the sensible thing: you search for studies. You find four trials, all reporting a real benefit, and you think — that's settled, four out of four. But four out of four is exactly what you'd see if the six trials that found nothing had been thrown away. You weren't shown a sample of the evidence. You were shown the flattering slice of it, and the unflattering slice never made it onto a page. The literature you can search is not the research that was done.

Two leaks, same direction

The first leak is the file-drawer problem. A trial that finds a clear effect feels like a finding worth submitting; a trial that finds nothing feels like a failure and gets shelved. Journals compound it — they have historically preferred to print positive, novel results over "we tried, nothing happened." The studies don't vanish because anyone lied. They vanish because nobody bothered to publish a null. The result is the same: the visible evidence drifts upward, away from the truth.

The second leak is funding. Across drugs, devices, and nutrition, industry-sponsored studies are markedly more likely to report results that favour the sponsor's product than independently funded studies of the same question. This rarely takes the form of fraud. It works through quieter levers: picking a weak comparator, choosing the dose and outcome most likely to win, and selective outcome reporting — measuring ten things, publishing the two that looked good, and "spinning" a flat result as promising in the abstract.

Work one, then finish one

Worked: A funnel plot is the standard detector for this. Plot every study's effect size against its precision: big, precise studies near the top, small, noisy ones spread wide at the bottom. With no bias the cloud is a symmetric inverted funnel — small studies scatter both ways around the true effect. When the bottom-left corner is empty, as above, that's the tell: the small studies that found no or negative effects exist but were never published. The asymmetry is the shape of the file drawer. Pair that with the funding pattern — sponsor-funded trials far likelier to report a favourable result, often via the quiet levers in the table — and a glowing literature can rest almost entirely on what you can't see.

Your turn: A supplement's website lists five glowing studies. All five were funded by the maker; there are no independent replications. How much should that move you? (Heavily discount it. You have a dense conflict of interest stacked on top of a likely file drawer of unpublished nulls — the published five may be the only ones that "worked," and the people who paid for them chose what to report. Treat it as close to no evidence until an independent group reproduces it.)

Why this matters

Supplements and "wellness" products are the single place where this damage is worst. There's no requirement to register a supplement trial or publish its result, so the file drawer runs deep; and the studies that do exist are overwhelmingly paid for by the seller. Imagine you're standing in the aisle holding a bottle that promises "clinically studied" joint relief, with a page of citations on the box. Before you spend the money, ask the two questions that cut through almost all of it: who paid for these studies, and where are the ones that found nothing? If every citation traces back to the maker and no independent lab has reproduced it, the confident claim and the missing nulls are telling you the same thing — wait for better evidence.