09-15-2019, 03:12 AM

From HERE

By chance I’ve been looking at the Palmer Drought Severity Index (PDSI) and the Palmer Modified Drought Severity Index (PMDI) lately. I’ve used data from the NOAA CLIMDIV

Mssrs. Christy and McKitrick are correct that Palmer drought have high Hurst exponents. I have found the same in the CLIMDIV data. The CLIMDIV data for the PMDI is monthly from 1895, with 1495 monthly records (often expressed as N = 1495). The Hurst exponent of that CLIMDIV PMDI data is 0.86.

However, when adjusted for autocorrelation the effective N for that dataset is not 1495. Instead, the effective N is equal to 8. And with only 8 independent data points, statistical significance is … well … elusive.

The problem is that as an article in Nature magazine was headlined, “Nature Is Naturally Trendy”. Any dataset with a high Hurst exponent will naturally have many more trends than we’d expect from independent random data. And many climate datasets, including temperature and sea level datasets, have high Hurst exponents.

I’ve been beating this drum for a while. In 2015 I wrote a post called “A Way To Calculate Effective N” in which I discussed and experimentally verified the effect of high Hurst exponents on the statistics of things like the Nilometer data. Let me shamelessly recommend that post as an overview of the subject and the size of the effect of high Hurst exponents on statistical significance.

My thanks to the authors for highlighting what I see as a very important and frequently overlooked issue in the statistics of climate. It’s a hard nettle to grasp because it shows that many claimed trends, when adjusted for autocorrelation, are not even approaching statistical significance.

w.

By chance I’ve been looking at the Palmer Drought Severity Index (PDSI) and the Palmer Modified Drought Severity Index (PMDI) lately. I’ve used data from the NOAA CLIMDIV

Mssrs. Christy and McKitrick are correct that Palmer drought have high Hurst exponents. I have found the same in the CLIMDIV data. The CLIMDIV data for the PMDI is monthly from 1895, with 1495 monthly records (often expressed as N = 1495). The Hurst exponent of that CLIMDIV PMDI data is 0.86.

However, when adjusted for autocorrelation the effective N for that dataset is not 1495. Instead, the effective N is equal to 8. And with only 8 independent data points, statistical significance is … well … elusive.

The problem is that as an article in Nature magazine was headlined, “Nature Is Naturally Trendy”. Any dataset with a high Hurst exponent will naturally have many more trends than we’d expect from independent random data. And many climate datasets, including temperature and sea level datasets, have high Hurst exponents.

I’ve been beating this drum for a while. In 2015 I wrote a post called “A Way To Calculate Effective N” in which I discussed and experimentally verified the effect of high Hurst exponents on the statistics of things like the Nilometer data. Let me shamelessly recommend that post as an overview of the subject and the size of the effect of high Hurst exponents on statistical significance.

My thanks to the authors for highlighting what I see as a very important and frequently overlooked issue in the statistics of climate. It’s a hard nettle to grasp because it shows that many claimed trends, when adjusted for autocorrelation, are not even approaching statistical significance.

w.

“I would rather have questions that can’t be answered than answers that can’t be questioned.” Richard P. Feynman.