The Meaning of P-Value in Statistics
The “P-value” indicates the probability that random chance could have produced statistical results at least as extreme as those observed. P-values cannot be below zero or above one.
The so-called soft sciences, such as sociology or psychology, generally use the following scale:
- “p < 1%” is “highly significant”;
- “1% < p < 5%” is “significant”;
- “p = 5% or more” is “not significant”, although “p < 10%” might still be cited as worth considering.
The “hard sciences” such as particle physics, often use a “5-sigma” significance test. The p-value “p=3%” corresponds to “3 sigma”; but “5 sigma” is approximately “p=0.000057”.
The Question of Correlation versus Causation
Even when two sets of scores are correlated significantly, the question of “cause and effect” may still be unsettled. For example, did higher education provide greater science comprehension, or do people with an innate aptitude for science comprehension do well at school and achieve a higher education?
Finding a correlation only states that the two scores are related, and does not determine cause or effect. On the other hand, if two scores are not correlated, then there is little point to asking whether one caused the other.
One danger in statistics is that the decision to plot one score on the horizontal x-axis implies that this is the cause, and the other score is the effect.
Understanding Prof. Kahan’s Report
First, Prof. Kahan reports a highly significant positive correlation between higher education and science comprehension; as well as a highly significant negative correlation between religiosity versus science comprehension. These results seem reasonable, and lend credence to Prof. Kahan’s data and methodology.
His third finding fits with popular impression that “conservative Republican” views are negatively correlated to science comprehension. With “p=0.03”, the result is significant. However, with “r=-0.05”, there is either little correlation or little variation in science comprehension scores for this group.
The fourth finding was “surprising”. However, Prof. Kahan points out that the r-value is trivial for both the third and fourth finding. As well, with “p=0.05”, the significance drops just below the threshold of “significant”.
The same respondents were used in both the third and fourth analyses. Prof. Kahan simply used a different indicator from the one trove of data.
Republicans vs. Tea Party
Let’s assume that only people with “conservative Republican” views would self-identify as “part of the Tea Party movement”. This author would draw one conclusion, that those in the Tea Party movement scored higher for science comprehension than Republicans outside the Tea Party.
However, although the findings demonstrated a small link between Tea Party and science knowledge, neither the third finding nor the fourth can support an argument that one finds statistically-significant greater science comprehension among those in the Tea Party movement.
Kahan, Dan. Some data on education, religiosity, ideology, and science comprehension. (2013). The Cultural Cognition Project at Yale Law School. Accessed October 18, 2013.
Kopan, Tal. Eureka! Tea partiers know science. (2013). Politico.com. Accessed October 18, 2013.
Weisstein, Eric W. Correlation Coefficient. MathWorld–A Wolfram Web Resource. Accessed October 18, 2013.
Weisstein, Eric W. Significance. MathWorld–A Wolfram Web Resource. Accessed October 18, 2013.