If reported as a determinant factor, then this is p-hacking.Īs a point of interest. Although 1% appears small, the analyst has actually dredged each of the demographic fields until he/she has stumbled upon what is actually an anomaly. But the reality is that the analyst has made a crucial mistake. there’s a 1-percent chance that this trend is an anomaly)Ī p-value appears impressive. The p-value associated is measured at 0.01 (i.e.We happen to find that those with kids between 5-7 are three times more likely to take up an offer than the rest of the population.We then look back at the demographic information (let’s say 300 demographic fields) on all 100,000 individuals to see whether anything looks predictive of the customer taking up an offer.A proportion take up the offer (let’s say 1,000).We run a campaign offering new cell-phone contracts to a large group of individuals (let’s say 100,000).P-hacking is the exercise of fishing through data looking for correlations that have a p-value (below your threshold) and reporting this correlation as statistically significant (without assessing causality). However, in recent times statisticians have made note that many studies that site “highly reliable” p-values (usually a p<0.05) have been subject to a dubious activity of “p-hacking” or data dredging. Now when we observe a study (medical trial, crime statistics, voting stats), we should always look out for the p-value. Essentially here, it would be related to size of sample relative to population and the size of the trend (percentage break-down of votes). Where a small set of observations are used to determine/infer how a population voted, the p-value indicates how significant a conclusion may be. You may have seen this in election poll surveys The challenge comes into play when the observations from this sample are extrapolated across the entire population. Leaving statistical jargon aside, we use the p-value to understand how confident we should be when observing a sample. To understand p-hacking, let’s look first at the p-value? Today I’d like to extend this conversation into a topical matter: p-hacking, also known as data fishing. It is part of a blog series on cognitive biases and logical fallacies that data analysts should avoid.

A year ago, I published an article about motivated reasoning and how that can damage the data analytics process.

Data dredging correlations