Equivalent of R's of cor.test in Python

Question

Is there a way I can find the r confidence interval in Python?

In R i could do something like:

cor.test(m, h)

    Pearson's product-moment correlation

data:  m and h
t = 0.8974, df = 4, p-value = 0.4202
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.6022868  0.9164582
sample estimates:
      cor 
0.4093729

In Python I can calculate r (cor) using:

r,p = scipy.stats.pearsonr(df.age, df.pets)

But that doesn't return the r confidence interval.

Rarotonga

Answer 1

Here's one way to calculate confidence internal

First get the correlation value (pearson's)

In [85]: from scipy import stats

In [86]: corr = stats.pearsonr(df['col1'], df['col2'])

In [87]: corr
Out[87]: (0.551178607008175, 0.0)

Use the Fisher transformation to get z

In [88]: z = np.arctanh(corr[0])

In [89]: z
Out[89]: 0.62007264620685021

And, the sigma value i.e standard error

In [90]: sigma = (1/((len(df.index)-3)**0.5))

In [91]: sigma
Out[91]: 0.013840913308956662

Get normal 95% interval probability density function for normal continuous random variable apply two-sided conditional formula

In [92]: cint = z + np.array([-1, 1]) * sigma * stats.norm.ppf((1+0.95)/2)

Finally take hyperbolic tangent to get interval values for 95%

In [93]: np.tanh(cint)
Out[93]: array([ 0.53201034,  0.56978224])

Misfile

Thank you, that solves it. I wonder why (and if) statsmodels and/or scipy don't already provide this. – Rarotonga

Well, I was surprised too or I haven't searched the stack well enough. – Misfile

Any updates on this? Ideally there should be a one-liner scipy function to compute this as opposed to the eight-liner method you provided above. – Blesbok