From Tiers to Percentiles: Introducing a Big New Change to Criticker

[BadCosmonaut]

i think the site would have to be a mind-reader to map such things automatically, there's no way of knowing simply by looking at an individual's ratings alone. my suggestion would simply be that every user has the opportunity to manually determine their own 50th percentile should they so wish, this would essentially solve the problem. i'm not a programmer so i don't know how hard that would be to implement, but i think it'd improve TCI (and therefore PSI) accuracy significantly, even if it'd be a little difficult to explain as a feature (i don't think the intelligence of criticker's userbase should be underestimated though).

Yeah, I agree. I'm sure there's a way to make it guess, but the guessing could pretty easily make it less accurate than not guessing in the first place. I agree some kind of manual mapping is a good option, but the question is how to implement it in a way that's more accurate than it is now and practical so that all users can understand what to do. Mapping only the 50th percentile would probably not be too hard for most people (although I'd be worried most people would just pick 50/100 by default which could defeat the purpose of the whole mapping system, since in reality most peoples' 50th percentile is probably somewhere else). While I think mapping the 50th percentile would make it more accurate than it is now, I think it stands to reason that if mapping one 'tier' makes it more accurate, then mapping more tiers would continue to make it more accurate. Where to draw the line then? And I think the answer is to make sure the system stays easy to understand and use for users.

In any case, I'd be really interested to see if this is something that could be tested behind the scenes somehow (like how they tested the current system and found it's more accurate than the old 10 tier system). Take 50 users that have rated a bunch of the same movies, ask them where their 50th percentile is, then do whatever comparisons can be done to see if the accuracy goes up or down.

If it goes up, then the question is how much more accurate can it get beyond that by having users manually map even more, while also keeping the mapping system easy to use and understand for users. It's already kind of confusing for some people as it is now.

[AFlickering]

all fair points. i recall someone else suggesting this, but i think it could be explained as an extension of the current manual quip system, because while that's merely cosmetic it's the same basic idea. anyway, i do think it's in the spirit of criticker to give users as much agency over the meaning of their ratings as possible, and aside from increased accuracy i also believe it would make criticker even more distinct from typical rating sites, so i do hope this gets considered eventually. there's clearly quite a bit of demand for it based on how many people have echoed these suggestions over the years.

[fabiovisnadi]

An alternative would be a species of form in which the users would answer to more accurate questions such as "The rating 60 to me means a) A filme in my personal canon b) a film I really enjoy c) a film that despite some flaws I still consider good d) a bad film, etc (not necessarily these answers and this kind of questions but as a way of translating into words and perceptions a rating system, as a mean of finding an equivalent between a numeric verdict and a verbal one.

[kyvetti]

To be honest I wouldn't be too worried about the differences in scales and watching habits because the system as it is already considers this to some effect.

No matter what percentage, the tier/percentage of the movie is still comparable to other movies, this movie being better/worse than that movie, and it is mirrored in PSIs ("this movie will be in your 40th percentile, but since you watch only movies you are likely to like, that's still pretty high")

Also, if the differences are really strong, so will be the TCI: a person who watches only movies he probably enjoys and a person who wades through all kinds of rubbish are likely to have such different rankings that TCI becomes so large those persons won't seriously affect each other. Because TCI is a congregate of numerous rankings, it does not simply mirror liking the same films but also to some extent is affected by similar viewing habits.

For users with a significant number of rankings (above some hundreds) I'd trust more the actual data they are presenting than self-defined "averages".

[AFlickering]

No matter what percentage, the tier/percentage of the movie is still comparable to other movies, this movie being better/worse than that movie, and it is mirrored in PSIs ("this movie will be in your 40th percentile, but since you watch only movies you are likely to like, that's still pretty high")

you're right that a person can respond to a 40th percentile PSI by saying "since i watch only movies i'm likely to like, that's still pretty high". the trouble is, PSIs are generated based on a hierarchy of TCIs, and the TCI hierarchy is generated based on the assumption that everyone likes the movies in their 40th percentile roughly the same amount. the problem isn't that an individual can't interpret the meaning of a PSI in context with their own individual scale, but that their PSI is often being based on the wrong TCIs.

Also, if the differences are really strong, so will be the TCI: a person who watches only movies he probably enjoys and a person who wades through all kinds of rubbish are likely to have such different rankings that TCI becomes so large those persons won't seriously affect each other. Because TCI is a congregate of numerous rankings, it does not simply mirror liking the same films but also to some extent is affected by similar viewing habits.

this is kind of my whole point though: TCI's are made higher or lower by something that's irrelevant to taste, and ideally that wouldn't be the case. i have acquaintances with middling TCIs who clearly have more compatible tastes with me than some of my top TCIs do, but solely because of different viewing habits they affect my predicted scores less than they ought to (and vice-versa).

[livelove]

this is kind of my whole point though: TCI's are made higher or lower by something that's irrelevant to taste, and ideally that wouldn't be the case. i have acquaintances with middling TCIs who clearly have more compatible tastes with me than some of my top TCIs do, but solely because of different viewing habits they affect my predicted scores less than they ought to (and vice-versa).

You always hit the nail on its head. Thank you, AFlickering.

you're right that a person can respond to a 40% percentile PSI by saying "since i watch only movies i'm likely to like, that's still pretty high". the trouble is, PSIs are generated based on a hierarchy of TCIs, and the TCI hierarchy is generated based on the assumption that everyone likes the movies in their 40% percentile roughly the same amount. the problem isn't that an individual can't interpret the meaning of a PSI in context with their own individual scale, but that their PSI is often being based on the wrong TCIs.

again spot-on, AFlickering!
Talking about the 40% percentile somewhat obfuscates the problem, though, as 40% is not all toooo bad ...
The problem becomes more apparent, when high scores are translated to very low percentiles:

user: getshort78 ranks only films he likes.
His 121 rankings range from 87-100.
His worst ranked film, Sleepy Hollow, has a score of 87 which translates to 1% percentile.
How meaningful is it to assign such a low percentage value (1%) to such a high score (87) ?
Particularly when the user obviously thinks Sleepy Hollow is a very good movie.
Heck, he even says exactly that in his mini-review: "Very good movie."

This is far from an isolated case:

Depending on the user, the bottom 20% (or 10%, or 1%) of "poorly rated" films can stretch from 0 to 60 points or even above.
I'll just pick a random film and show you some scores with corresponding percentiles to illustrate my point:

• Akaasha: 70 = 27%
• evansjoy: 54 = 6%
• denzilac: 73 = 16%
• Loop: 60 = 11%
• Yottabyter: 81 = 12%
• draven88: 85 = 21%
• miyoung1: 68 = 19%
• aralic: 70 = 12%
• kadirke: 67 = 21%
• Wolfenrocks: 68 = 14%
• itachi18: 60 = 10%
• rakanichu: 65 = 7%
• minato: 75 = 4%
• sgrannis: 55 = 10%
• punica: 79 = 14%
• holland_49: 30 = 0%
• jrjunker: 70 = 6%
• athfrith: 65 = 12%
• NerdScience: 74 = 22%
• svink: 50 = 9%
• macgyver0: 50 = 4%
• mrkwst22: 80 = 10%
• Coffey: 60 = 9%
• dushanj: 52 = 3%
• amoxus: 50 = 3%
• zephyrblade: 59 = 5%
• talen: 70 = 20%
• Matthias99: 80 = 23% "Pretty bad. Woo needs to figure out how to not suck again."
• aliencowboy: 60 = 12%
• sgtpeppr: 75 = 11%
• menacedee: 73 = 7%
• Tide: 75 = 21%
• whaupwit: 70 = 14%
• gokce2604: 60 = 10%
• Murat O: 70 = 18%
• wizardctp: 70 = 18%
• blazewide: 71 = 13%

[kyvetti]

this is kind of my whole point though: TCI's are made higher or lower by something that's irrelevant to taste, and ideally that wouldn't be the case. i have acquaintances with middling TCIs who clearly have more compatible tastes with me than some of my top TCIs do, but solely because of different viewing habits they affect my predicted scores less than they ought to (and vice-versa).

Ideally yes, but as I was saying, TCI mirrors also watching habits and not just taste.
Finding some means of easily finding compatible users based on taste alone would be more difficult, probably impossible. Some smidgen of it could be garnered by comparing a bunch of very popular films that most people have ranked and comparing rankings based on only those, but, well, it will probably say something whether one prefers Pulp Fiction, Forrest Gump or Lord of the Rings but not much about more idiosyncratic parts of the taste...(might be interesting to run a test account copying just the rankings of most popular films and see if TCIs are wildly different)

Meanwhile ranking inflation portrayed by people giving rankings like 70 or 80 to films they think suck tells me many people also wouldn't be any good at assigning any arbitrary midpoints or other milestones.

[livelove]

ranking inflation portrayed by people giving rankings like 70 or 80 to films they think suck tells me many people also wouldn't be any good at assigning any arbitrary midpoints or other milestones.

Ok. But what about people who rank movies they deem "very good" at 87 and have them in their 1% percentile ?

[paulofilmo]

i hope people unwilling to experiment with a range of scores will get confused and frustrated

[kyvetti]

ranking inflation portrayed by people giving rankings like 70 or 80 to films they think suck tells me many people also wouldn't be any good at assigning any arbitrary midpoints or other milestones.

Ok. But what about people who rank movies they deem "very good" at 87 and have them in their 1% percentile ?

Well, I'd say it's anyway nigh impossible to generate reasonable estimations for people who rank only films they think are very good or better, sometimes one just can't win. Then it is better to just admit that feature won't work for people who do that.

Anyway, now that it came up, a possible metric of interest would be comparison of all rankings and those of only the most mainstream films, say, 250 or 500 films with most rankings...that doesn't sound like it's too difficult thing to code but if it demands too much resources, that I don't know...
It would give some image to what direction one deviates from "films everyone watches", whether one picks other films with more care or watches lot of whatever, and comparing ratings from a select group of films gives a better image of shared tastes, and for that group it makes sense to pick films that are seen by a lot of people, for more data.
Throwing the idea up there.