Learn pathway
How do I judge a market claim?
Samples, uncertainty, bias, testing and replication without statistical theatre.
- 01Reading the evidence · beginnerRead guide
Why sample size matters
A perfect-looking rate can still be weak evidence when it rests on very few observations.
- 02Reading the evidence · beginnerRead guide
A result needs a baseline
A result becomes meaningful only when it is compared with what would ordinarily happen under a relevant definition.
- 03Reading the evidence · beginnerRead guide
Uncertainty is part of the result
A result is more useful when you can see the range of values still compatible with the available evidence.
- 04Reading the evidence · intermediateRead guide
Backtesting without time travel
A backtest replays an idea on old data. It is useful only when the replay respects what could have been known at the time.
- 05Reading the evidence · beginnerRead guide
Probability
Probability represents uncertainty on a scale from impossible to certain under a defined model or evidence base. It describes possible outcomes, not a hidden guarantee about one event.
- 06Reading the evidence · beginnerRead guide
Chance and randomness
Randomness is variation not predictably explained by the information or model being used. It can produce streaks, clusters and convincing patterns even when no durable signal exists.
- 07Reading the evidence · intermediateRead guide
What is a p-value?
A p-value asks how unusual the observed result would be if a particular chance-only model were true.
- 08Reading the evidence · intermediateRead guide
What does statistical significance mean?
Statistical significance means a result crossed a pre-chosen threshold under a particular test. It does not mean important or profitable.
- 09Reading the evidence · intermediateRead guide
What is a confidence interval?
A confidence interval shows a range of values compatible with the estimate under a stated statistical method.
- 10Reading the evidence · beginnerRead guide
False positives
A false positive occurs when a test signals an effect or condition that is not present under the relevant truth definition. The rate depends on thresholds, prevalence and the testing process.
- 11Reading the evidence · beginnerRead guide
False negatives
A false negative occurs when a test fails to detect an effect or condition that is present. Small samples, noisy measurement or strict thresholds can make detection unlikely.
- 12Reading the evidence · intermediateRead guide
Multiple testing
Multiple testing occurs when many hypotheses, groups, thresholds or outcomes are examined. Even valid individual tests can produce an impressive-looking winner by chance across the wider search.
- 13Reading the evidence · intermediateRead guide
What is overfitting?
Overfitting happens when a rule learns the accidents in old data so closely that it struggles with new data.
- 14Reading the evidence · intermediateRead guide
Look-ahead bias
Look-ahead bias lets information unavailable at a historical decision time influence that decision or its measured outcome. It gives the backtest a form of time travel.
- 15Reading the evidence · intermediateRead guide
Data snooping
Data snooping is repeated exploration of the same dataset until a favourable pattern is found, often without carrying the full search into the reported uncertainty.
- 16Reading the evidence · beginnerRead guide
In-sample testing
In-sample testing evaluates an idea on data available for developing, selecting or tuning it. It shows fit to the workshop material, not independent validation.
- 17Reading the evidence · intermediateRead guide
What does out of sample mean?
Out-of-sample testing applies a frozen rule to data that did not help create or tune it.
- 18Reading the evidence · intermediateRead guide
Prospective testing
Prospective testing records a rule, inputs and expectation before future outcomes exist, then attaches those outcomes without rewriting the original record.
- 19Reading the evidence · intermediateRead guide
Correlation and causation
Correlation shows that variables move together under a sample; causation means changing one would change the other under specified conditions. Shared causes, selection and chance can create correlation without causation.
- 20Reading the evidence · intermediateRead guide
Survivorship bias
Survivorship bias occurs when analysis includes entities that remain visible while omitting those that failed, closed or left the dataset. The survivors can make history look safer or stronger.
- 21Reading the evidence · intermediateRead guide
Selection bias
Selection bias arises when inclusion in a sample depends on factors related to the outcome, making the observed group unrepresentative of the intended population.
- 22Reading the evidence · intermediateRead guide
Publication bias
Publication bias occurs when results are more likely to become visible because they are positive, striking or statistically significant. The available record then overstates effects.
- 23Reading the evidence · beginnerRead guide
Confirmation bias
Confirmation bias is the tendency to seek, interpret and remember information in ways that support an existing belief while discounting disconfirming evidence.
- 24Reading the evidence · intermediateRead guide
Regression to the mean
Regression to the mean is the tendency for an extreme noisy observation to be followed by one closer to the typical level, even without a causal intervention.
- 25Reading the evidence · intermediateRead guide
Independent observations
Observations are independent when knowing one does not change the relevant probability distribution of another under the model. Market events often share firms, dates or shocks and are not fully independent.
- 26Reading the evidence · intermediateRead guide
Effect size
Effect size describes the magnitude of a difference or relationship in meaningful or standardised units. It complements uncertainty and helps distinguish detectable effects from useful ones.
- 27Reading the evidence · advancedRead guide
Statistical power
Statistical power is the probability that a specified test rejects its null when a specified alternative effect is true. It depends on effect size, sample, variability and threshold.
- 28Reading the evidence · intermediateRead guide
Robustness
Robustness is the degree to which a conclusion survives reasonable changes in assumptions, definitions, samples and methods. It is accumulated evidence, not one favourable alternative specification.
- 29Reading the evidence · intermediateRead guide
Replication
Replication repeats a study or test using the same method, independent implementation, new data or a new setting. Each form checks a different source of error.
- 30Reading the evidence · beginnerRead guide
What is a hypothesis?
A hypothesis is a precise, testable statement about a relationship or outcome. It turns a broad observation into something that data could support, weaken or leave unresolved.
- 31Reading the evidence · intermediateRead guide
Falsifiability
Falsifiability means a claim permits some possible evidence to count against it. A claim that explains every outcome after the fact cannot be meaningfully tested.
- 32Reading the evidence · beginnerRead guide
Signal and noise
Signal is repeatable information relevant to the question; noise is variation that obscures or imitates it under the chosen model. The separation is uncertain and context-dependent.
- 33Reading the evidence · intermediateRead guide
Win rate and profitability
Win rate is the proportion of positive outcomes; profitability depends on the size of wins and losses, costs, exposure and sequence. A strategy can win often and still lose money.
- 34Reading the evidence · beginnerRead guide
Mean, median and outliers
The mean adds values and divides by their count; the median is the middle ordered value. Outliers and skew can separate them, so each describes a different centre.
- 35Reading the evidence · beginnerRead guide
Reading a distribution
A distribution shows which values occurred or are possible and how frequently or plausibly they appear. Shape reveals spread, skew, clusters and tails hidden by one average.
- 36Reading the evidence · intermediateRead guide
Sampling error
Sampling error is the variation between a sample estimate and the population quantity caused by observing only part of the population. It differs from bias and measurement error.
- 37Reading the evidence · intermediateRead guide
Pre-registering a test
Pre-registration records the hypothesis, data, exclusions and analysis plan before outcomes are examined. It separates planned confirmation from later exploration and makes deviations visible.
- 38Reading the evidence · intermediateRead guide
Data quality and missing information
Data quality is fitness for a specific use across completeness, accuracy, consistency, timing, provenance and meaning. Clean formatting does not establish that observations represent the intended facts.