The Bad Science Audit

1. •  The Bad Science Audit

     The Concept

     Journalists rarely read the scientific papers they write about. They read the “Press Release” written by a university PR department. This is why you see headlines like “Glass of Wine Equals Hour at Gym.”

     To find the truth, you must bypass the article and audit the Abstract of the original paper. You don’t need a PhD; you just need to check for three specific “Crimes.”

     ---

     LAB EXERCISE 1: The “Relative Risk” Trap

     The Scenario: A headline screams, “New Study Shows Bacon Increases Cancer Risk by 20%!” The Objective: Prove this is statistically meaningless.

     The Mission:

     1. Find the “Absolute Risk”: You must find the original number of people who got cancer.

     2. The Calculation:

        • Group A (No Bacon): 5 out of 100 people got cancer. (5% Risk).

        • Group B (Bacon): 6 out of 100 people got cancer. (6% Risk).

        • The Difference: The absolute difference is 1 person (1%).

     3. The Media Trick:

        • 6 is 20% larger than 5. So the media reports a “20% Increase.”

        • The Reality: Your risk went from “Tiny” to “Still Tiny.”

     4. Action: Whenever you see a percentage increase (“Double the risk!”), ask: “Double what?” (Double a 0.0001% risk is still 0.0002%).

     ---

     LAB EXERCISE 2: The “Surrogate Outcome” Hunt

     The Scenario: A new drug claims to “Reduce Heart Attacks.” The Objective: Check if they actually measured heart attacks, or if they cheated.

     The Mission:

     1. Open the study (or the detailed news report).

     2. Search for the phrase “Primary Endpoint” or “What was measured.”

     3. The Audit:

        • Real Outcome: Did they count how many people died or had a heart attack?

        • Surrogate Outcome: Did they measure Blood Pressure, Cholesterol, or Artery Thickness?

     4. The Verdict: Just because a drug lowers cholesterol (the Surrogate) does not mean it stops heart attacks (the Real Outcome). It might lower cholesterol but cause liver failure. If they didn’t measure the event itself, the claim is unproven.

     ---

     LAB EXERCISE 3: The “Cherry Pick” (p-Hacking)

     The Scenario: A study claims “Green Jelly Beans Linked to Acne.” The Objective: Find the 19 failed tests they hid.

     The Mission:

     1. Look at the “Methods” section of the paper.

     2. Ask: “How many variables did they test?”

        • Did they only test Green Jelly Beans?

        • Or did they test Red, Blue, Yellow, Purple, and Orange beans too?

     3. The Math: If you test 20 different colors with a standard “p-value” of 0.05, one of them will appear statistically significant by pure chance.

     4. The Verdict: If they tested 20 things but only reported on the 1 that “worked,” that is not science. That is a lottery.

     ---

     Hands-On: The “Abstract Check”

     Task: Find a health news story on CNN or BBC today.

     1. Find the link to the actual study (usually at the bottom or hyperlinked in the first paragraph).

     2. Read the Conclusion in the Abstract.

     3. Compare: Does the scientist’s conclusion match the journalist’s headline?

        • Scientist: “We observed a minor correlation in mice models.”

        • Journalist: “CURE FOUND!”

     4. Result: You will find they rarely match.