Research methods · Measurement

Does it generalize? External validity

The study is real. The athletes really did get stronger, the dose was measured, the eight weeks happened. None of that is in question. The question is whether their result is your result. They were two decades younger, training hard, eating to gain. You're cutting calories on a drug that blunts appetite and may eat into muscle. A finding can be flawless inside its own walls and still tell you almost nothing about the person reading it. The gap between "true in the study" and "true for me" has a name, and learning to see it is most of what separates a careful reader from a credulous one.

The study describes a narrow slice of the world

Every trial recruits a specific, bounded group: a particular age, sex, health status, and training level, given a particular dose in a particular form for a particular length of time. The result is true for that slice. You, and everyone else who reads the headline, are the wide world the slice was carved out of. External validity is just the honest question of how far the slice stretches — and the answer is usually "less far than the headline implies."

Four dimensions almost always differ between the study and you. Run down them like a checklist before you believe a claim applies:

Work one, then finish one

Worked: A label says "clinically studied to support joint comfort," citing a trial that used 5 grams of an ingredient daily. You flip the bottle: each capsule contains 0.5 grams, and the label suggests one a day. The trial dose was ten times what the product delivers. Even if the study were flawless, it cannot support this product as sold — you'd have to take ten capsules a day to reach the dose that was actually tested, which no one does and the label never says. The citation is real and irrelevant at once. Dose mismatch alone breaks the chain from evidence to pill.

Your turn: A large trial in older Spanish adults found a Mediterranean diet cut heart events. Does that transfer to a young South Asian on a calorie cut? (Maybe partly, but you can't assume it. Population differs — age, and South Asians carry different cardiometabolic and genetic baseline risk. Baseline diet differs — the benefit was measured against a typical Spanish diet, not a calorie-restricted one. What you'd check: was the effect ever tested in a population like yours, what was the baseline being improved on, and does "Mediterranean" even mean the same foods in your context.)

Why this matters

Dose-mismatch and wrong-population quietly invalidate a huge share of supplement and wellness claims aimed at the general public. A brand can cite a genuine study, with a straight face, while the product delivers a fraction of the studied dose to a body nothing like the people studied — and both moves are invisible if you only read the headline. The same trap runs through podcast advice ("works wonders" — in whom?) and forum testimonials (one person, one body, no control). When you're deciding whether to spend money or change your routine, external validity is the cheapest filter you own: trace the claim back to its slice of the world, then ask whether you're standing in that slice. Usually you're not, and that alone settles it.