Jay's Blog
Math, Teaching, Literature, and Life
July 25, 2022
This is the third part of a three-part series on hypothesis testing.
Hypothesis testing is central to the way we do science, but it has major flaws that have encouraged widespread shoddy research. In this essay we consider methods that can help us draw better conclusions, and avoid the pitfalls of hypothesis testing. We start with some smaller and more conservative ideas, which basically involve doing hypothesis testing _better_. Then we'll look at more radical changes, taking the focus away from hypothesis tests and seeing the other ways we can organize and contribute to scientific knowledge.
Comments
May 24, 2022
This is the second-part of a three-part series on hypothesis testing.
Today we'll look at the way we do hypothesis testing in practice, and how it tends to fail. Modern researchers use hypothesis testing as a tool to develop knowledge, but it's really a tool for making decisions, and so it encourages us to draw strong conclusions from weak evidence. It also encourages us to view studies that don't reject the null hypothesis as failures, which leads even honest and dedicated researchers to do shoddy research, producing "statistically significant" results that can't be reproduced.
Comments
March 31, 2022
This is the first part of a three-part series explaining what hypothesis testing is and how it works. In this essay I'll talk about the way hypothesis testing developed historically, in two rival schools of thought. I'll explain how these two methodologies were originally supposed to work, and why you might (or might not) want to use them.
Comments
February 02, 2022
The replication crisis is a major problem in medicine and social science; we know that a huge fraction of the published literature is outright wrong. But in math we don't seem to have a similar crisis, despite reasonably frequent minor errors in published papers. Why not, and what can this tell us about the fields that are in crisis?
Comments
November 28, 2021
A back-of-the-envelope cost-benefit analysis tells us that taking ivermectin for covid might have positive expected value. If we follow that logic to its conclusion, we wind up taking twenty different supplements and this seems like it can't be wise. Resolving this apparent conflict exposes some of the deep flaws in how we often think about rationality and Bayesian reasoning.
A response to a piece by Scott Alexander at Astral Codex Ten.
Comments
Support my writing at Ko-Fi