From Tiers to Percentiles: Introducing a Big New Change to Criticker

[livelove]

you have to rate at least 1 movie for each point in your distribution scale.

Ok, but that is not what the above discussion was about (and also not a particularly common scenario).
What was discussed was most users watch only films they like and thus only rate films they like, often leading to rating scales ranging e.g. between 50-100 with no or very few ratings below 50 (as these are the very films, experienced cineasts mostly manage to avoid). Criticker's percentile system treats this situation in the very same way as Criticker's tier system (just with higher granularity), as was pointed out in the discussion above:

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad.

[BadCosmonaut]

Ok, but that is not what the above discussion was about.
What was discussed was most users watch only films they like and thus only rate films they like, often leading to rating scales ranging e.g. between 50-100 with no ratings below 50.

I'm curious, do you know of any users who have no ratings below 50 and who also consider 50 to be mediocre or above?

Also, I have a hard time believing that most users have no ratings below 50 and intend 50 to not be the bottom of their personal scale. How sure are you that most users are like that?

Criticker's percentile system treats this situation in the very same way as Criticker's tier system (just with higher granularity), as was pointed out in the discussion above:

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad.

My example covers this scenario just fine. 20% of rated movies are poorly rated. That will give them their fair share of bad tiers. In this example, the new system works.

[livelove]

I'm curious, do you know of any users who have no ratings below 50 and who also consider 50 to be mediocre or above?

I said "no or very few". And yes, I think there are many who consider 50 medicore. I don't see where exactly you are going with this.

Also, I have a hard time believing that most users have no ratings below 50 and intend 50 to not be the bottom of their personal scale. How sure are you that most users are like that?

Again, I said "no or very few". 50 was just an example, really. Could be 40, if you prefer. All I wanted to say is that there are users with quite few (if any) votings below a certain point, because that's precisely the kind of movies folks avoid watching. And yes, I think there are many users like that, because I have personally seen lots of threads about this very issue (2 of which I have linked to above, and it also came up here in the discussion multiple times, not really by chance).

Criticker's percentile system treats this situation in the very same way as Criticker's tier system (just with higher granularity), as was pointed out in the discussion above:

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad.

My example covers this scenario just fine. 20% of rated movies are poorly rated. That will give them their fair share of bad tiers. In this example, the new system works.

Nobody said, Criticker does not "work"! The question some of the other users pondered here was HOW it works, what the possible shortcomings could be and whether there might be a better solution.

I think what you are missing is, that the "new system" (percentiles) does not handle skewed distributions fundamentally differently than the old tier system. In fact, save for the higher granularity, it deals with it in the very same manner.

Depending on the user, the bottom 20% (or 10%, or 1%) of "poorly rated" films can stretch from 0 to 60 points or even above.
I'll just pick a random film and show you some scores with corresponding percentiles to illustrate my point:

• Akaasha: 70 = 27%
• evansjoy: 54 = 6%
• denzilac: 73 = 16%
• Loop: 60 = 11%
• Yottabyter: 81 = 12%
• draven88: 85 = 21%
• miyoung1: 68 = 19%
• aralic: 70 = 12%
• kadirke: 67 = 21%
• Wolfenrocks: 68 = 14%
• itachi18: 60 = 10%
• rakanichu: 65 = 7%
• minato: 75 = 4%
• sgrannis: 55 = 10%
• punica: 79 = 14%
• holland_49: 30 = 0%
• jrjunker: 70 = 6%
• athfrith: 65 = 12%
• NerdScience: 74 = 22%
• svink: 50 = 9%
• macgyver0: 50 = 4%
• mrkwst22: 80 = 10%
• Coffey: 60 = 9%
• dushanj: 52 = 3%
• amoxus: 50 = 3%
• zephyrblade: 59 = 5%
• talen: 70 = 20%
• Matthias99: 80 = 23% "Pretty bad. Woo needs to figure out how to not suck again."
• aliencowboy: 60 = 12%
• sgtpeppr: 75 = 11%
• menacedee: 73 = 7%
• Tide: 75 = 21%
• whaupwit: 70 = 14%
• gokce2604: 60 = 10%
• Murat O: 70 = 18%
• wizardctp: 70 = 18%
• blazewide: 71 = 13%

[BadCosmonaut]

I think it actually does handle skewed distributions fundamentally different than the old system. I could be wrong, but I explained how so above. The problem of skewed distribution was that the old system would merge (into the same tier) movies that were very differently rated in order to squeeze them into the same 10% chunk.

For example, if a user had just 5 (out of 1,000) movies rated below a 50, then the old system would probably put all those movies into the same Tier 1. This could be a problem since those 5 movies could have been very differently rated. Like the spread could be 1, 10, 20, 30, 49. That's a huge gap between those scores to put them all into one tier. Yet that's what would happen under the old system.

I don't think that problem can happen under the new system, since every single different rating possible will have its own tier. Using my example above, the movie rated 1 would have its own tier. The movie rated 10 would have its own. 20 would have its own. 30 its own. And 49 its own. Since there are 100 possible tiers, each of those would get their own. They wouldn't be merged into one.

The question that remains then is do any of the 100 tiers merge into something like the 10% chunks for purposes of comparison. If so, then there very well could be a problem with the new system, but I haven't seen an example of this yet, and I'm not sure if only MPowell could answer this.

I'm curious, do you know of any users who have no ratings below 50 and who also consider 50 to be mediocre or above?

I said "no or very few". And yes, I think there are many who consider 50 medicore. I don't see where exactly you are going with this.

You did not originally say "no or very few." You originally just said no, as can be seen in my original quote. I'm guessing I quoted you before you edited your post. I think it's a little disingenuous to know that you edited your post then to act like I misquoted you...

As far as where I was going with that question, you said:

Ok, but that is not what the above discussion was about.
What was discussed was most users watch only films they like and thus only rate films they like, often leading to rating scales ranging e.g. between 50-100 with no ratings below 50.

Since you said most users have no ratings below 50, I asked for an example. There very well may be some users like that. I'm curious how many though, since you also said most users are like that. Where I was going with this is that most users are not like that. Most users do have some low ratings. Specifically, most users have ratings that cover the full spectrum of their personal scale. If this is true, then I'm still not seeing the problem with the current system other than the potential problem I discussed above.

I'm trying to understand what you're saying the problem is, preferably with a specific example. I can't really even say if I agree or disagree until then because we may actually agree.

[AFlickering]

i feel like this is getting needlessly complex so let me try to give a clear example:

let's say user A is someone who exclusively watches films with a high psi and user B is a film critic who has to watch every godawful new movie that hits theaters. this means user A enjoys 80% of the films he watches, and user B only enjoys 35% she watches. they go to see a film together, and they both like the movie exactly the same amount (let's say mildly): they even give it the same rating of 60. however, because user A mostly watches films he likes a LOT, that rating only lands in his 35th percentile, whereas user B watches so much garbage that a rating of 60 lands in his 80th percentile. so the site thinks there's a 45 percentile difference between their tastes regarding this film, when in reality they had an identical response to it. this would happen even if they're using a 100 point scale and have rated films every possible rating on that scale.

[livelove]

I'm guessing I quoted you before you edited your post.

No. If that were the case, you would see the "last edited by [...]" footnote.

I think it's a little disingenuous

I never reply to ad-hominem stuff.
Just a quote from yourself to someone else (which apparently doesn't apply to yourself):

What practical result do you think can come from this conversation?
Also, it's rare where a worthwhile discussion happens after personal attacks/name calling.

I will reply to you regardless.

you edited your post then to act like I misquoted you...

Please stop your conspiracy theories, ok. I never said you misquoted me, let alone on purpose. You just missed my edit and I tried to point you to what you have missed, alright.

I think it actually does handle skewed distributions fundamentally different than the old system.

What practical result do you think can come from this conversation? You keep on repeating this on and on and don't seem willing to try to think through my explanations at all. Maybe we should just leave it at that. You think, that I'm talking nonsense and I think that there are misconceptions in your reasoning.

This discussion has gotten off its rails and is becoming a distraction.
We should either stop it or concentrate on the facts, which I'll try one more time below.

The problem of skewed distribution was that the old system would merge (into the same tier) movies that were very differently rated in order to squeeze them into the same 10% chunk.

I'm sorry, but we have been over this a couple of times already. I cannot explain it any better than I already did.
Nobody and nothing tries to "squeeze" anything. It's just math.

there have been a lot of users who would like to "assign" certain scores to certain tiers, which is just not possible, because... it's math. It's not "configurable".

Simply put:
The old tier system was a 10% percentile system.
The new system is a 1% percentile system.
You get finer granularity, and thus higher accuracy. But that's really it. Other than that, it's the same system.
mpowell even said as much:

Because they're so similar on the back-end, we're allowing tiers and percentiles to co-exist

both systems can put very high scores into very low tiers or percentiles. Have you even looked at my examples above??
e.g:
• mrkwst22: 80 = 10%
• dushanj: 52 = 3%

Both of these scores would very likely be in tier 1 (in the old system). Now they are in the 3% and 10% percentile (corresponding to tier 0.3 and tier 1, if you will). Why are you unable to see, that you get finer granularity (1% steps instead of 10% steps), but both systems deal with skewed distributions in the very same manner. If you don't see that by now, I'll give up. Someone else will have to jump in.

For example, if a user had just 5 (out of 1,000) movies rated below a 50, then the old system would probably put all those movies into the same Tier 1. This could be a problem since those 5 movies could have been very differently rated. Like the spread could be 1, 10, 20, 30, 49. That's a huge gap between those scores to put them all into one tier. Yet that's what would happen under the old system. I don't think that problem can happen under the new system, since every single different rating possible will have its own tier.

Not only do you still fail to see, that fabiovisnadi, AFlickering, cke and myself were discussing something else entirely, what you say ("every single different rating possible will have its own tier") is just plain wrong:
• If you have > 100 ratings, than some ratings obviously have to share the same percentile.
• And no, "every single different rating possible will have its own tier" would only be possible if 1 = 1%*, 2 = 2%, 3 = 3% and so on ... but if you look at my long ranking list above, you'll see that this very obviously not the case.
(*actually 1 = 0%, but I'll try to keep it simple)

Using my example above, the movie rated 1 would have its own tier. The movie rated 10 would have its own. 20 would have its own. 30 its own. And 49 its own. Since there are 100 possible tiers, each of those would get their own. They wouldn't be merged into one.

That's also wrong.
The user in your example has 1000 rankings: 1, 10, 20, 30, 49 and 995 higher rankings.
In the old tier system rankings 1-49 (1, 10, 20, 30, 49) all fall into tier 1.
In the new percentile system rankings 1-49 (1, 10, 20, 30, 49) all fall into the 0% percentile.

The question that remains then is do any of the 100 tiers merge into something like the 10% chunks for purposes of comparison. If so, then there very well could be a problem with the new system, but I haven't seen an example of this yet, and I'm not sure if only MPowell could answer this.

I have offered you repeated demonstrations to the contrary, which you all ignore. Yes, go ahead and ask mpowell, since you don't take it from me.

If you don't trust my maths, why don't you just verify it by recalculating ?

Since you don't take it from me, maybe you'll take it from an actual user:
Matthias99: https://www.criticker.com/profile/Matthias99
total ratings: 730
8x score of 30 ==> in 2% percentile
1x score of 40 ==> also in 2% percentile

The worst part is, that you are not only wrong, but completely missing the point.
In fact, we didn't even discuss that matter, but something else entirely. Nevermind.

I'm curious, do you know of any users who have no ratings below 50 and who also consider 50 to be mediocre or above?

Yes, quite a few. But it gets even better, this user rates from 87-100:
https://www.criticker.com/rankings/getshort78

you said:

Ok, but that is not what the above discussion was about.
What was discussed was most users watch only films they like and thus only rate films they like, often leading to rating scales ranging e.g. between 50-100 with no ratings below 50.

You know what, I'll take that back. You *DO* misquote me on purpose. The first time it could have been an oversight on your behalf. But afterwards I told you, and now you misquote the very same thing again. I said "no OR very few ratings below 50"

Ok, but that is not what the above discussion was about. What was discussed was most users watch only films they like and thus only rate films they like, often leading to rating scales ranging e.g. between 50-100 with no ratings below 50.

Since you said most users have no ratings below 50, I asked for an example.

I also never said that! Please read carefully. I said "most users watch only films they like and thus only rate films they like, often leading to [skewed] rating scales e.g. ..." e.g. means example given, which means that I give 1 specific example as a random example for a skewed rating scale. I never said that most users have no ratings below 50. Big difference!
And ... Jesus! Again: I didn't say "no ratings below 50", I said "... or very few". How many times are you actually going to repeat that incorrectly on purpose !?
Just to let you know: If you misquote me again, I'll stop replying.

And I already gave you plenty of examples, prior to this post as well as in this post (e.g. user getshort78 above).

I'm curious how many though, since you also said most users are like that.

Repeating it doesn't make it true.

most users are not like that. Most users do have some low ratings.

If you keep on misquoting me, that's not my fault. I said "... or very few ratings below 50" which is equal to "some low ratings".

Specifically, most users have ratings that cover the full spectrum of their personal scale.

you are just guessing

If this is true, then I'm still not seeing the problem with the current system other than the potential problem I discussed above.

Jesus! Nobody (definitely not me at any rate) said there were a problem!
We were all very nuanced in our discussion. We just mused aloud about the challenge of imbalanced voting distributions. mpowell will be the first to confirm that this is indeed a challenge and that there is no perfect solution for this and that he probably has found the best one of all imperfect solutions.

I'm trying to understand what you're saying the problem is

Again, we never said the new system has a problem (since it works just like the old one, just with higher precision and thus higher prediction accuracy). We wondered out aloud how to deal with users who mostly watch films they like (fabiovisnadi estimates that ~80% of films are ranked this way).

preferably with a specific example.

I already gave you plenty. Here is one of them:
user: getshort78 ranks only films he likes.
His 121 rankings range from 87-100.
His worst ranked film, Sleepy Hollow, has a score of 87 which translates to 1% percentile.
How meaningful is it to assign such a low percentage value (1%) to such a high score (87) ?
Particularly when the user obviously thinks Sleepy Hollow is a very good movie.
Heck, he even says exactly that in his mini-review: "Very good movie."

The old tier system puts his score-87-film into his tier 1.
The new percentile system puts his score-87-film into the 1% percentile.
That goes to show, that both systems work in the same manner in that regard.
What the system does, is to "normalize" / "unskew" / "distribute evenly" the scores across the tiers/percentiles as best as the granularity (10 vs 100) permits, in an attempt to translate user'A scores into "what user A really wanted to say IN THE LANGUAGE OF USER B (i.e. in the context of user B's voting scheme)".

Now the question is whether getshort78's score of 87 for Sleepy Hollow "really means" 1% (as Criticker assumes) or really means "a very good movie with a score of 87" as the user himself says.

THAT's what was up for debate (between the 3 above-mentioned users and myself).

The problem is, that you can find (many) examples where "what Criticker assumes" really reflects better what the user actually tries to express by his scores (e.g. score of 58, 3% percentile, and user says it's a very bad movie).

The challenge is to find a method that works in BOTH cases.

[livelove]

i feel like this is getting needlessly complex so let me try to give a clear example:
let's say user A is someone who exclusively watches films with a high psi and user B is a film critic who has to watch every godawful new movie that hits theaters. this means user A enjoys 80% of the films he watches, and user B only enjoys 35% she watches. they go to see a film together, and they both like the movie exactly the same amount (let's say mildly): they even give it the same rating of 60. however, because user A mostly watches films he likes a LOT, that rating only lands in his 35th percentile, whereas user B watches so much garbage that a rating of 60 lands in his 80th percentile. so the site thinks there's a 45 percentile difference between their tastes regarding this film, when in reality they had an identical response to it. this would happen even if they're using a 100 point scale and have rated films every possible rating on that scale.

(quoted for awesomeness)

spot-on! thank you, you hit the nail on its head.

A similar example:
• create a second account, e.g. with username AFlickering-test
• copy all your rankings from AFlickering to AFlickering-test
• delete the bottom half of AFlickering-test's rankings
• result: the upper half of AFlickering-test's rankings will be distributed to much lower percentiles, ALTHOUGH they are still the SAME films, with the SAME score, and ranked/liked by the SAME person (you).

This goes to show, that your scores don't mean what you want them to mean,
but depend on what you have watched before and after.

[BadCosmonaut]

i feel like this is getting needlessly complex so let me try to give a clear example:

let's say user A is someone who exclusively watches films with a high psi and user B is a film critic who has to watch every godawful new movie that hits theaters. this means user A enjoys 80% of the films he watches, and user B only enjoys 35% she watches. they go to see a film together, and they both like the movie exactly the same amount (let's say mildly): they even give it the same rating of 60. however, because user A mostly watches films he likes a LOT, that rating only lands in his 35th percentile, whereas user B watches so much garbage that a rating of 60 lands in his 80th percentile. so the site thinks there's a 45 percentile difference between their tastes regarding this film, when in reality they had an identical response to it. this would happen even if they're using a 100 point scale and have rated films every possible rating on that scale.

This example makes sense. Thanks. That's all I was asking for, is an explanation and example of the problem so I could understand it. If criticker assumes in all cases that something like 35th percentile is different than 80th percentile between two users, then TCI will be less accurate. My understanding of the new system was that it would map each used percentile between two users, so that, for example, my 'i liked it mildly' percentile would be mapped to each other user's 'i liked it mildly' percentile, regardless of how many movies either of us has seen and regardless of how different our personal scales are. If that happens, then TCI wouldn't be negatively affected in your example. If this kind of mapping doesn't happen, then I agree this could be a problem, and my question then is whether this kind of mapping should happen (and how to implement it).

I'm not sure I want to inflame the rest of the conversation anymore, but I have a couple of notes. The "last edited by" doesn't always appear under certain circumstances. I'm not sure exactly what those circumstances are, but I believe it doesn't always appear either when a comment is edited very quickly after being posted and/or if a comment is edited before a different user loads that page of that thread. As for what practical result can come from this discussion-there are a few things. One is if there is a flaw in the system, it can be explained so someone like me who didn't see it can understand it. That's why I've been replying, so I can understand. Then beyond that, if there is a problem it can be discussed and hopefully improved upon or even solved. This is different from the board game thread since nothing was going to come from that conversation other than just an academic discussion as to whether board games and video games should be split on this site. There was no practical purpose to that discussion from my side of the argument.

[livelove]

The "last edited by" doesn't always appear under certain circumstances. I'm not sure exactly what those circumstances are

The edit timestamp always appears when any posting (save a thread's last one) is edited. -- Please, let's end this.

my 'i liked it mildly' percentile would be mapped to each other user's 'i liked it mildly' percentile

How do you define the "liked it mildly" percentile? Can you put a number to it? Or how else do you determine it mathematically?

If I give you a user, can you tell me what his/her "liked it mildly" percentile is?
How about users (like getshort78) who only watch/rank films they love (and thus don't have a "liked it mildly" percentile) ?

[AFlickering]

i feel like this is getting needlessly complex so let me try to give a clear example:

let's say user A is someone who exclusively watches films with a high psi and user B is a film critic who has to watch every godawful new movie that hits theaters. this means user A enjoys 80% of the films he watches, and user B only enjoys 35% she watches. they go to see a film together, and they both like the movie exactly the same amount (let's say mildly): they even give it the same rating of 60. however, because user A mostly watches films he likes a LOT, that rating only lands in his 35th percentile, whereas user B watches so much garbage that a rating of 60 lands in his 80th percentile. so the site thinks there's a 45 percentile difference between their tastes regarding this film, when in reality they had an identical response to it. this would happen even if they're using a 100 point scale and have rated films every possible rating on that scale.

This example makes sense. Thanks. That's all I was asking for, is an explanation and example of the problem so I could understand it. If criticker assumes in all cases that something like 35th percentile is different than 80th percentile between two users, then TCI will be less accurate. My understanding of the new system was that it would map each used percentile between two users, so that, for example, my 'i liked it mildly' percentile would be mapped to each other user's 'i liked it mildly' percentile, regardless of how many movies either of us has seen and regardless of how different our personal scales are. If that happens, then TCI wouldn't be negatively affected in your example. If this kind of mapping doesn't happen, then I agree this could be a problem, and my question then is whether this kind of mapping should happen (and how to implement it).

i think the site would have to be a mind-reader to map such things automatically, there's no way of knowing simply by looking at an individual's ratings alone. my suggestion would simply be that every user has the opportunity to manually determine their own 50.00th percentile should they so wish, this would essentially solve the problem. i'm not a programmer so i don't know how hard that would be to implement, but i think it'd improve TCI (and therefore PSI) accuracy significantly, even if it'd be a little difficult to explain as a feature (i don't think the intelligence of criticker's userbase should be underestimated though).