Intuitive Understanding of Statistical Significance

People often rely on intuitive statistical reasoning in their daily lives. We observe events, form hypotheses, and eventually accept them as truths based on our intuitive models, even if we don’t use technical terms like “statistical” or “significance.” Consider this scenario:

Imagine you’re driving through town and stop at a traffic light near the square. You let out a sigh of exasperation because you anticipate that if this light turns red, you'll face a series of red lights ahead. You wish you had received a green light instead.

In this case, you have created a mental database from your past experiences and developed an “If-Then” hypothesis. As you gather more experiences (red and green lights), you confirm your initial hypothesis as a truth. You won’t reconsider this “truth” unless a significant number of experiences contradict your original hypothesis.

In this intuitive model, any data point that confirms your hypothesis contributes to its statistical significance. Thus, your intuitive model gains importance through this confirmation.

However, there are significant issues at play. One major philosophical question I have discussed elsewhere involves the raven paradox. Additionally, the scientific interpretation of statistical significance often differs sharply from our intuitive understanding.

Scientific Statistical Significance

When it comes to scientific methods, intuition frequently falls short. One of the trailblazers of modern statistical methodology was Ronald Aylmer Fisher. The contemporary statistical method can be summarized in the following steps:

1. Establish the null hypothesis.
2. Conduct an experiment and collect data.
3. Assess the likelihood of observing the results as exceptions (or extremes) to the null hypothesis, assuming it holds true. This probability is known as the p-value.
4. If the p-value falls below a predetermined threshold (5% was Fisher’s preferred level), the results are considered statistically significant, leading to the rejection of the null hypothesis. Conversely, a higher p-value indicates that the null hypothesis remains unchallenged.

The initial step involves framing the null hypothesis, which highlights a critical distinction between intuitive and scientific statistical significance. In our intuitive models, confirming events are marked as statistically significant.

In contrast, the scientific approach requires that the data be sufficiently inconsistent with the negation of the hypothesis (the null hypothesis) for it to be labeled statistically significant.

If this concept seems abstract, consider this example: Suppose I claim I can control the sun's rising. Using intuitive reasoning, if you observe the sun rising consistently, you might deem my hypothesis statistically significant.

However, employing the scientific method, the null hypothesis would state that I cannot influence the sun's rising. You could test this by asking me not to make the sun rise. If the sun rises anyway, the null hypothesis stands.

A More Practical Scenario

Let’s say you are testing a cream designed to protect individuals from radio waves and hypothesize that it will be statistically significant in its effects.

The corresponding null hypothesis would be: the cream has no effect on people. To test this hypothesis, you conduct an experiment with 100 participants, splitting them into two groups—50 receive the cream and 50 receive a placebo. You then measure the radio wave-shielding effectiveness for both groups.

It is not enough for the cream group to show better shielding results than the placebo group; it is also essential that fewer placebo participants exhibit comparable or superior shielding (as indicated by the p-value). Only then can the results be deemed statistically significant, meaning the null hypothesis is rejected.

While I’ve simplified this example by not delving into natural shielding probabilities or assumed distributions, it suffices for the purpose of this discussion.

Having covered Fisher's scientific method of null-hypothesis significance testing, let’s turn our attention to the challenges associated with this approach.

Challenges with Statistical Significance

Most challenges arise at what I refer to as interface points. A typical statistical workflow appears as follows:

As illustrated, the experimental and statistical treatment of the issue at hand are largely objective protocols. However, complications arise when formulating the null hypothesis and interpreting the results of statistical significance.

Addressing the Issues with Statistical Significance

To begin tackling the challenges outlined, we must first focus on the interface points. There is no single solution to these problems.

When confronted with “statistically significant” results, we must critically evaluate the formulation of the null hypothesis and the linguistic interpretation of “statistical significance”.

While there can be flaws in the statistical methods themselves, the interface points pose a more pressing issue—they play with human emotions. A dramatic interpretation could provoke an unwarranted reaction from people (ten times a minuscule number remains a minuscule number).

In conclusion, the term “statistical significance” conveys nothing about the quantitative significance of the results. It merely indicates that we are detecting something worthy of our attention.

Reference and credit: Jordon Ellenberg.

Further reading: Consider exploring How to Perfectly Predict Improbable Events? and The New Industrial Revolution Is Here.

If you wish to support my work as an author, please consider contributing on Patreon.