[new] Prediction Accuracy Index (PAI)

[livelove]

The 2 main goals of the criticker math are:
- finding other people with similar movie taste (low TCI)
- making accurate predictions for how much you will like a given film (PSI)

This literally begs for introducing some measure of how accurate those predictions are.

Criticker users are surely interested in how good their predictions have become and
will appreciate seeing their accuracy rise with time as they rank more and more films
(which is an additional incentive to rank more films, thus improving criticker's health).

Seeing that you like 3-letter acronyms (PSI, TCI), I named this measurement the
Prediction Accuracy Index (PAI).

One could also call it Mean Prediction Error (MPE).

It compares the probable score to the actual score.

I just added each film's difference between its PSI and my ranking and then divided the sum by the number of films.

Code: Select all

         PSI  score  diff
film 1:   14     0    14 
film 2:   37    15    22
film 3:   34    10    24
film 4:   56    65     9
film 5:   89    85     4
film 6:  100   100     0
film 7:   32    25     7
film 8:   91   100     9
film 9:   36    35     1
film10:   24    50    26
=========================
                     116

Code: Select all

                     116 / 10 films = 11.6 
                                      11.6/(maxscore–minscore) = 11.6/(100-0) = 11.6%

As an example, above are my 10 last ranked films, which have a PAI of 11.6
Which means that based on these 10 films, my PSIs are off by about 12 points on average.
(The user's PAI would of course be calculated based on ALL his rankings, not just 10 films.)

This is a pretty good (low) value, given that I have only imported 66 movie rankings into my profile thus far, therefore the PAI can be expected to get better (lower) as more rankings are added.

How do you like that idea ?

I think rather than as an absolute number, it would be more meaningful to express the PAI as a percentage of the user's scoring scale, spanning from the user's lowest ranked film to the highest ranked one (e.g. 20-95).
My scale ranges from 0 to 100, so my PAI is 11.6 %

Unless you can think of a better place, I'd display the PAI in the own userprofile and also in other userprofiles:

[livelove]

hi @mpowell.
I think you haven't read my suggestion back at that time, but apparently you came up with the very same idea and implemented it now.
Could you read my suggestion above and say how it matches up to what you have done ?
thank you.

[mpowell]

We definitely read it back in 2015, but actually developed this idea independently. It came as we were testing the accuracy of the new switch to Percentiles... once we had the basic testing framework developed, it was a no-brainer to work it into the sponsor features.

Thanks for making that suggestion! Even if we weren't directly referencing your idea, it might have influenced us subconsciously!

[livelove]

@mpowell:
How does your implementation compare to my suggestion above ?
Are there any differences, and if so, what are they ?

[mpowell]

@mpowell:
How does your implementation compare to my suggestion above ?
Are there any differences, and if so, what are they ?

I suppose they're pretty similar, as it's simply a comparison of score to psi. Unfortunately, we just don't have time to do a thorough analysis into how the implemented system differs from your suggestion. You can check that out yourself, though, since all our calculations are right out there in the open (if you're a sponsor, that is! )

[livelove]

I suppose they're pretty similar […] Unfortunately, we just don't have time to do a thorough analysis into how the implemented system differs from your suggestion.

Comparing your image below to my codeblock above, I'd say it's exactly the same.

[livelove]

My suggestion has an additional element, though, that you could add to your implementation:
the Prediction Accuracy Index (PAI).

explanation: see OP
in short: It's a measure of how accurate your predictions are and is calculated as follows:

Code: Select all

PAI = ∅ difference / (maxscore-minscore)

maxscore: score of highest ranked film
minscore: score of lowest ranked film

In other words, it takes what you call the "average difference" and turns this absolute value into a relative one, namely as percentage of the user's full voting range.
(If user A's ratings span from 0-20 and user B's from 50-95, their avg.difference is divided by 20 or 45, respectively, see OP).

This allows for direct comparison between users (something users fancy bigtime) regardless of their ranking scale.

For this reason I'd suggest adding the PAI to all userprofiles (see screenshot above) and other places you deem helpful, for example in this form:
• PAI: 4.7%, or
• prediction accuracy: 4.7%

(I'd recommend spelling it out, as there are already enough acronyms.
Adding even more could become confusing and deterring for new users).

You could limit the PAI to sponsors, but for the following reasons I'd recommend giving it to everyone, as in this case it would seem to be in your own interest:

Criticker users are surely interested in how good their predictions have become and will appreciate seeing their displayed prediction accuracy rise with time as they rank more and more films, which is an additional incentive to rank more films, thus improving criticker's health.

I imagine it would also improve Criticker's prediction accuracy overall, since users, to whom you display a bad (high) PAI, will strive to get a better PAI in ways that are beneficial to Criticker overall (e.g. by trying to get a more balanced vote distribution, rethinking their voting scale, ranking more bad films, etc.)

And as you said in the other thread (I'll paraphrase a little): “it'd be fun to know how accurate Criticker is” for each user.

[livelove]

Taking into account the intelligent ideas by other users, I'd like to make my suggestion visually easily understandable right off the bat in order to help make the site more appealing and accessible to new users.

changes:

• tidied up profile entries
• added percentage values (for direct comparison between users with different ranking scales!)
• removed PAI acronym (for the aforementioned reasons)
• added tooltips (as explanation for new users)
• linkified numbers (you'd need to find meaningful URLs)

Screenshots:

your own profile:


other profile (tooltip #1):


other profile (tooltip #2):