From Tiers to Percentiles: Introducing a Big New Change to Criticker

[livelove]

This chart:
"Criticker's recommendation system is designed to work just as well with rating data that is not "perfectly distributed".

This may be the design intention, but I disagree that it works.

Criticker algorithm:
advantage: severe and lenient voters can be compared to each other
disadvantage: since we avoid bad movies, our lowest ranked movies (even with excellent scores nominally) are treated by Criticker as "terrible"

comparing scores at face value:
advantage: the scores you attribute as a user are respected
disadvantage: severe and lenient voters cannot be compared to each other

The upshot seems to be:
The ideal system would use an algorithm that respects users votes while authentically translating different voting systems to one another. I don't have an idea yet of how to accomplish this.

When I started here, I very much liked Criticker's way of "stretching out" your scores within a 0-10 scale (old tier system) or 0-100 (new percentile system). This is certainly a very unique feature among movie ranking sites.

But the more I think about it, the more it bothers me, that our scores are not respected (e.g. movies with excellent scores are treated by Criticker as "bad" or "terrible"). Criticker pretends to know better than ourselves what our scores mean. This is a little patronizing.

Now I get that this is not done with bad intent. And I even concede that when I see users ranking a film at 70 while their mini-review says the film is terrible, I understand the need for some kind of intervention. On the other hand, there are very experienced movie lovers who meticulously ponder what scores to assign and taking that away from them just doesn't feel right either.

I don't know, what the solution is, unfortunately. But I think some kind of compromise is needed, i.e. an algorithm, which is more respectful of users' rankings.

[AnttiR]

I think when counting TCI's, instead of comparaison of percentiles you should use Spearman's rank correlation coefficient.
I do not watch bad movies, but some other peoples watch. With current system, person who has identical taste but who has watched more bad movies, will have a very different TCI.

[livelove]

AnttiR, I think your posting is the most important posting I ever read here on Criticker.

With current system, person who has identical taste but who has watched more bad movies, will have a very different TCI.

... although the TCI should be zero.

You make an excellent point, given that Criticker's purpose is measuring identical taste.

I read the wiki articles on Spearman's rank correlation coefficient but have only limited understanding of how it works (my maths isn't sufficient). How could it be put to use for Criticker ?

[TMDaines]

So I switched from tiers to percentiles last week and it doesn’t look like my TCIs and PSIs have been regenerated. Less than 1000 films have PSIs. How long does this take? I presume it is hours rather than days.

[AnttiR]

AnttiR, I think your posting is the most important posting I ever read here on Criticker.

With current system, person who has identical taste but who has watched more bad movies, will have a very different TCI.

... although the TCI should be zero.

You make an excellent point, given that Criticker's purpose is measuring identical taste.

I read the wiki articles on Spearman's rank correlation coefficient but have only limited understanding of how it works (my maths isn't sufficient). How could it be put to use for Criticker ?

Hi, thanks for writing me a PM as I did not followed this thread actively.

It is not difficult to implement Spearman's rank correlation coefficient in comparaision of two users. This procedure has 3 steps

-Calculate intersection of films both users A and B have seen
-Number films of this intersection by order of user A, and order of user B.
-Calculate Spearmans rank correlation coefficient between these two numbersings.

[livelove]

Hi, thanks for writing me a PM as I did not followed this thread actively.

It is not difficult to implement Spearman's rank correlation coefficient in comparaision of two users. This procedure has 3 steps

-Calculate intersection of films both users A and B have seen
-Number films of this intersection by order of user A, and order of user B.
-Calculate Spearmans rank correlation coefficient between these two numbersings.

thanks for your explanation. That sounds VERY interesting, I'm intrigued ...

Could you provide a concrete example (with numbers) showing how to calculate, or would this be too difficult ?

[spacefloh]

Thanks to the Team Criticker.

With the "new" percentile calculations I now understand myself better in terms of emotions I had with my movie ratings. Often I felt with a mediocre movie (i.e. 50 points) disappointed and thought that it was horrible. Now, with the percentile ratings I did understand that I was just watching way to less bad movies, and why these mediocre movies get a ~10 percentile rating.

Watching fewer bad movies, watching more good ones was the intention to come to criticker in the first place (And ofc. not to watch movies twice accidentially).
Therefore, goal achieved.


Even if you work with mathematics yourself, you may not be aware of what each number means. This is solved for me and the rates now.
So, thanks again ( @mpowell ).

[ambient_gf]

I am so frustrated because it seems I need to start watching more bad films in order for my percentiles / tiers to make sense. Everyone complains of this. Nobody seems to have a solution. Maybe let us set our own tiers? I have no idea. I prefer criticker over letterboxd and have stuck to criticker for years but it's that one feature that seems to be pronounced the older I get and the less of a chance I even have to simply stumble upon a bad / mediocre film during a social gathering or on television. The decision to even watch the film I watch is based on the great possibility I might like them. I don't want to waste my time watching bad films. Yes, I can set my own descriptions and I can explain my own ratings in the bio but it is still pretty irritating. I just switched to percentiles after years of avoiding it and I think it made the problem even worse.

[iconogassed]

Your percentiles and tiers make sense as they are. They simply reflect your very particular approach to scoring and its relation to a perfectly even distribution of ratings. It's not a matter of being too generous or of having seen too many good movies. In fact, simply ranking more bad or mediocre films would work against your aims: note that, towards the higher end of the scale, you are much "stricter". This means that for every additional low score, the difference between your scores of 80+ and their corresponding percentiles would increase.

[livelove]

I feel you. It's a bit ironic. It seems we must do the very thing we wanted to avoid by joining Criticker:
watching bad films.

Watching fewer bad movies, watching more good ones was the intention to come to criticker in the first place

I am so frustrated because it seems I need to start watching more bad films in order for my percentiles / tiers to make sense. Everyone complains of this. Nobody seems to have a solution.
[…]
The decision to even watch the film I watch is based on the great possibility I might like them. I don't want to waste my time watching bad films. Yes, I can set my own descriptions and I can explain my own ratings in the bio but it is still pretty irritating.

Maybe the "Spearman's rank correlation coefficient" could solve the problem.
@AnttiR: If you are still here, maybe you could tell us more about it ?