Research methods · Putting it to work
Spotting weak studies & hype
Try this first
You have 60 seconds and one bold health claim in front of you: "clinically proven to boost energy 40%." Before reading on, write down what you'd check — and the order you'd check it in. Which question would you ask first?
Here's the freeing part: you almost never need to read the study. Most over-hyped claims collapse in the first ten seconds of a fixed checklist, the same way a flimsy chair gives out the moment you lean on the first leg. You don't inspect all four legs — you sit, it fails, you move on. The skill isn't deep reading. It's running a short triage in a fixed order and letting the claim fall at the first hurdle it can't clear.
The one idea
Run a fixed checklist, in order, and stop at the first failure. Humans or animals? Controlled or just observed? Hard outcome or a surrogate? Absolute numbers or a relative percent? Big and long enough? Independent or paid for? Most hype dies on the first one or two questions — you never reach the rest.
The six-question filter
Order matters. Cheaper, more decisive questions go first, so the weakest claims fail fastest. Each question has a "no" side that should make you stop and stay skeptical, and a "yes" side that lets the claim advance to the next rung. Survive all six and you've earned the right to actually read the paper.
| Ask | "No" means |
|---|---|
| Humans? | Mice, cells in a dish, or a worm — stop. It may never translate. |
| Controlled / randomized? | Just watched a group over time — can't separate cause from "healthy-user" habits. |
| Hard outcome? | A surrogate (a blood marker, a scan, self-reported "energy") stood in for what you care about. |
| Absolute effect shown? | Only a relative percent ("40% more") with no baseline to anchor it. |
| Big and long enough? | A handful of people for a few weeks — noise, not signal. |
| Independent funding? | The seller ran or paid for the study that conveniently flatters the product. |
Work one, then finish one
Worked: a tweet says "clinically proven to boost energy 40%!" Run the filter. Humans? The brand's own page links a study in mice — already wobbling; if you push past it, keep going. Controlled? No control group; everyone took the product and was asked how they felt — fails. Hard outcome? "Energy" is self-reported on a questionnaire, a textbook surrogate primed for the placebo effect — fails. Absolute? "40%" is a relative number with no baseline; 40% more than what? — fails. By now you've spent maybe twenty seconds and the claim has fallen at three separate hurdles. You never needed the funding question. Dismiss and move on.
Your turn: triage the headline "mouse study shows compound reverses aging." Where does it fall, and how far do you have to read? (First hurdle: it's a mouse study, so "humans?" already fails. Stack on a surrogate outcome — a biomarker of aging, not lifespan — and a giant leap from one rodent result to "reverses aging" in people. You stop at question one; the rest is hype piled on a finding that may never translate.)
Why this matters
This is the everyday tool — the one you actually reach for. Say you're staring at a $45 bottle of a "longevity" supplement that a podcaster swears by and a Reddit thread is buzzing about. You don't have time to read trials, and you shouldn't have to. You run the filter on the loudest claim: human trial or mouse study? randomized or just a survey? did anyone live longer, or did one blood marker move? a real effect size or a scary-sounding percent? funded by the company selling it? Sixty seconds later you either have a reason to look harder or you've saved yourself $45 and a daily pill that does nothing. Multiply that over every ad, thread, and podcast claim you meet in a year, and a fast, fixed filter is worth more than any single fact you could memorize.
Recall check · no peeking
- List the fast study-level red flags in the order you'd check them, and why that order saves time.
- Name the person-and-promotion red flags — the tells in who is making the claim, not the data.
- What is the single error every one of these weak claims shares?
Explain it back
In one line, name the universal tell: what does every over-hyped claim do with the evidence sitting underneath it?