Reading the evidence
Multiple testing occurs when many hypotheses, groups, thresholds or outcomes are examined. Even valid individual tests can produce an impressive-looking winner by chance across the wider search.
Make it click
One blind throw landing near the centre is unusual; hundreds of throws make at least one close hit far less surprising. Showing only the best throw hides the search.
Now put that into markets
Research matrices, feature screens and repeated parameter choices multiply opportunities for false positives. The family of tests is part of the evidence.
Family-wise and false-discovery controls answer different goals. Pre-registration, holdouts and replication reduce selection problems rather than merely adjusting a p-value.
Count populated cells, outcomes, filters and abandoned versions. Report whether correction was planned before results.
A correction cannot repair poor hypotheses or hidden exploration, and overly severe control can reduce useful discovery. Separate exploration from confirmation.
Takeaway
