Consequences of extreme values for Pearson's $r$ in this applied scenario

user1205901 07/04/2016. 2 answers, 81 views
correlation pearson-r

A friend and I are looking at ways of selecting forecasters based on past performance. My friend's preferred method is to select the forecasters who are most correlated with the truth, based on a Pearson correlation.

I've been trying to express to my friend some reasons why I think this method is a bit flawed, using the following example:

|  True value  |  Forecaster 1  |  Forecaster 2   |
|      5       |        6       |        51       |
|      10      |        9       |        99       |
|      20      |        21      |       201       |

Forecaster 1’s estimates are ±1 from the true value, while Forecaster 2’s estimates are ±1 from (true value × 100). The Pearson correlation between Forecaster 1 and the truth is .9897, while the Pearson correlation between Forecaster 2 and the truth is > .9999. Recall that according to my friend's method we're selecting the Forecaster who has the highest Pearson correlation with the truth. Thus in this example Forecaster 2 would be selected, even though Forecaster 2’s forecasts are wildly variable and inaccurate relative to Forecaster 1’s.

  1. Does my example illustrate a genuine problem with my friend's method?
  2. If so, how can I describe the problem in a more rigorous way? In technical terms, why does the Pearson correlation have this property?
  3. What kind of constructive solution can I offer? Is there some other type of obvious selection method (e.g. a different type of correlation) that would be more robust?
Nick Cox 07/04/2016
Search the forum for concordance correlation. Threads include……
Glen_b♦ 07/04/2016
In relation to your second point -- the Pearson correlation is location- and scale- invariant (up to sign). The problem is that accuracy is dependent on location and scale.

2 Answers

mdewey 11/21/2017.

Yes it does disclose a problem although your friend may have a different scientific question in mind (although it is hard to guess what that might be). Forecaster 3 who guesses 105, 110, 120 does even better I think you will find.

There are a host of options. In the medical field an influential paper for the case where you are comparing two measurements is by Bland and Altman available here entitled "Statistical methods for measuring agreement between two methods of clinical measurement". In the thirty years since publication it has been cited more than 38000 times according to Google.

You might also want to look at mean squared error

Ian_Fin 07/04/2016.

As far as I can see, the issue here is similar to the issue of consistency vs. agreement in the context of interrater reliability. Consider your true value as ratings offered by one rater (who happens to be very good...) and the forecasters as providing their own ratings.

Raters are consistent if, e.g., those people who are rated as higher than average by one rater are also rated similarly higher than average by another rater. Consistency can be assessed with correlations.

Raters are in agreement if the mean of one rater's set of ratings are similar to the mean of another rater's set of ratings. Agreement can be assessed with tests of differences between means.

Consistency and agreement are orthogonal. Two raters can be consistent, but not in agreement,

         |Rater 1|Rater 2|
Subject 1|   1   |   10  |
Subject 2|   2   |   20  |
Subject 3|   3   |   30  |
Subject 4|   4   |   40  |
Subject 5|   5   |   50  |
     Mean|   3   |   30  |    r = 1

and in agreement but not (really) consistent,

         |Rater 1|Rater 2|
Subject 1|   1   |   1   |
Subject 2|   2   |   5   |
Subject 3|   3   |   4   |
Subject 4|   4   |   2   |
Subject 5|   5   |   3   |
     Mean|   3   |   3   |    r = .1

or both, or neither (you can imagine those numbers for yourself).

Ideally, you want both consistency and agreement to be high. Both of these can be assessed using the intraclass correlation coefficient

Unless I'm missing something obvious, you could do something similar here between your true values and your forecasts. - Download Hi-Res Songs

1 The Chainsmokers

Beach House flac

The Chainsmokers. 2018. Writer: Andrew Taggart.
2 (G)I-DLE


(G)I-DLE. 2018. Writer: Riot Music Team;Harloe.
3 Anne-Marie

Rewrite The Stars flac

Anne-Marie. 2018. Writer: Benj Pasek;Justin Paul.
4 Ariana Grande

​Thank U, Next flac

Ariana Grande. 2018. Writer: Crazy Mike;Scootie;Victoria Monét;Tayla Parx;TBHits;Ariana Grande.
5 Clean Bandit

Baby flac

Clean Bandit. 2018. Writer: Jack Patterson;Kamille;Jason Evigan;Matthew Knott;Marina;Luis Fonsi.

Waste It On Me flac

BTS. 2018. Writer: Steve Aoki;Jeff Halavacs;Ryan Ogren;Michael Gazzo;Nate Cyphert;Sean Foreman;RM.
7 Imagine Dragons

Bad Liar flac

Imagine Dragons. 2018. Writer: Jorgen Odegard;Daniel Platzman;Ben McKee;Wayne Sermon;Aja Volkman;Dan Reynolds.
8 Nicki Minaj

No Candle No Light flac

Nicki Minaj. 2018. Writer: Denisia “Blu June” Andrews;Kathryn Ostenberg;Brittany "Chi" Coney;Brian Lee;TJ Routon;Tushar Apte;ZAYN;Nicki Minaj.
9 Brooks

Limbo flac

Brooks. 2018.
10 BlackPink

Kiss And Make Up flac

BlackPink. 2018. Writer: Soke;Kny Factory;Billboard;Chelcee Grimes;Teddy Park;Marc Vincent;Dua Lipa.
11 Diplo

Close To Me flac

Diplo. 2018. Writer: Ellie Goulding;Savan Kotecha;Peter Svensson;Ilya;Swae Lee;Diplo.
12 Rita Ora

Velvet Rope flac

Rita Ora. 2018.
13 Fitz And The Tantrums

HandClap flac

Fitz And The Tantrums. 2017. Writer: Fitz And The Tantrums;Eric Frederic;Sam Hollander.
14 Little Mix

Woman Like Me flac

Little Mix. 2018. Writer: Nicki Minaj;Steve Mac;Ed Sheeran;Jess Glynne.
15 Billie Eilish

When The Party's Over flac

Billie Eilish. 2018. Writer: Billie Eilish;FINNEAS.
16 K-391

Mystery flac

K-391. 2018.
17 Cher Lloyd

None Of My Business flac

Cher Lloyd. 2018. Writer: ​iamBADDLUCK;Alexsej Vlasenko;Kate Morgan;Henrik Meinke;Jonas Kalisch;Jeremy Chacon.
18 Backstreet Boys

Chances flac

Backstreet Boys. 2018.
19 Lil Pump

Arms Around You flac

Lil Pump. 2018. Writer: Rio Santana;Lil Pump;Edgar Barrera;Mally Mall;Jon Fx;Skrillex;Maluma;Swae Lee;XXXTENTACION.
20 Kelly Clarkson

Never Enough flac

Kelly Clarkson. 2018. Writer: Benj Pasek;Justin Paul.

Related questions

Hot questions


Popular Tags