Since I am a subscribed member and have access to the feature that shows Prediction Accuracy,
I decided to play around with the "Films in common minimum". I now have rated 1100 movies, so I assumed I could begin to lower it.
Prediction Accuracy shows 4 parameters of which the three of them got better after the change (median difference shows 10 whatever I set it to):
The average difference (between my the probability scores and my actual ratings).
The amount of "spot on" (where the PSI matched my actual ratings exactly)
The amount of "way-of" PSI's (more than 34 from my actual rating).
It was at 25% and tried to reduce it to 20. All parameters showed better results. But I wanted to know what the optimal "Films in common minimum" was, so I kept lowering the value, waiting for a point where it would start to give worse predictions. But the predictions got better and better the more I reduced it. At last I was where it couldn't get lower: 3%. Here I get the most precise predictions.
During this journey, 25>20>15>10>8>6>5>3 I wondered if I had misunderstood something? People in this post have written about optimal percentages being 25 or 13, so can 3% really be my optimal "Films in common minimum"?
I do not understand tpbradbury's post, if I could I might see that I am way-off with the 3% (he seems to correct BadCosmonaut who uses a method similar to mine).
I can give you the numbers:
Average difference has come down from 11.91 to 9.47
Spot-on has risen from 154 to 162
Way-off has gone down from 36 to 11 (admittedly that is the only big change)
All this show that criticker predicts better now, right?
My best TCIs were in the 25.000 range before
Now they are in the 18.000 range.
(my former best TCI has now become my #78)
And yes, it's true that lots of them have not watched that many movies - my best TCI has only seen 80 movies - but I have seen 33 of them! And of these we totally agree on 13 movies! Only one we totally disagree on.
So am I right that 3% seems to be the best in my case? (maybe because my choice of movies is not so typical?)
Optimal 'Films In Common Minimum'
-
oplars
- Posts: 48
- Your TCI: na
- Joined: Sun Jul 13, 2014 4:57 pm
-
geohawk
- Posts: 61
- Your TCI: na
- Joined: Sat Apr 05, 2008 5:50 am
Re: Optimal 'Films In Common Minimum'
As many people like to say when looking at something like this: "It Depends!"
It appears that your experience is not uncommon - that as you lower the minimum % required, that you do create possibilities for users that more closely match your own ratings to enter your TCIs. But, because they tend to have fewer rated movies, the pool of films that can be used to calculate the prediction accuracy gets smaller. At a minimum, this means that there is greater variability in Prediction Accuracy. What tpbradbury is concerned is that this method of improving Prediction Accuracy may also be creating a bias. This is probably correct: as we increase the minimum, the pool of users must have seen a greater number of the movies that we have seen. So we sacrifice "accuracy" by increasing the breadth of movies for which we can get PSIs.
For instance: at 10% (my old norm), I can get enough ratings that I can get a prediction for which TV series from 1970 I am most probably will like ("Mary Tyler Moore Show"). If I drop down to 3%, Criticker is not able to calculate a PSI for any of the qualifying entries in its database.
I think there is also risk of bias on the other side: as you increase your minimum, your TCIs will more likely have to fall into a set of users with large amount of ratings. These include film buffs, aggregators ("users" that represent averages from other sites, such as imdb, Rotten Tomatoes, or Movielens), and others. The ones that are most similar to you will be favoured because they are most likely to be in the Top 10 ratings used to calculate your PSI. But from this pool, you might find that it is harder to find one that has a TCI that more closely reflects your own interests.
I often wish that sites like this would give us a mechanism to encourage "discovery" and take us to movies that we are less likely to consider just on our own. Criticker doesn't give us many tools for this, but this, and the "popularity" filter can do a bit of the work here.
It appears that your experience is not uncommon - that as you lower the minimum % required, that you do create possibilities for users that more closely match your own ratings to enter your TCIs. But, because they tend to have fewer rated movies, the pool of films that can be used to calculate the prediction accuracy gets smaller. At a minimum, this means that there is greater variability in Prediction Accuracy. What tpbradbury is concerned is that this method of improving Prediction Accuracy may also be creating a bias. This is probably correct: as we increase the minimum, the pool of users must have seen a greater number of the movies that we have seen. So we sacrifice "accuracy" by increasing the breadth of movies for which we can get PSIs.
For instance: at 10% (my old norm), I can get enough ratings that I can get a prediction for which TV series from 1970 I am most probably will like ("Mary Tyler Moore Show"). If I drop down to 3%, Criticker is not able to calculate a PSI for any of the qualifying entries in its database.
I think there is also risk of bias on the other side: as you increase your minimum, your TCIs will more likely have to fall into a set of users with large amount of ratings. These include film buffs, aggregators ("users" that represent averages from other sites, such as imdb, Rotten Tomatoes, or Movielens), and others. The ones that are most similar to you will be favoured because they are most likely to be in the Top 10 ratings used to calculate your PSI. But from this pool, you might find that it is harder to find one that has a TCI that more closely reflects your own interests.
I often wish that sites like this would give us a mechanism to encourage "discovery" and take us to movies that we are less likely to consider just on our own. Criticker doesn't give us many tools for this, but this, and the "popularity" filter can do a bit of the work here.
-
oplars
- Posts: 48
- Your TCI: na
- Joined: Sun Jul 13, 2014 4:57 pm
Re: Optimal 'Films In Common Minimum'
I started pondering about this scenario once more, and decided to check with the new ChatGpt o1-preview for its perspective. I copy/pasted the last two posts and asked for its opinion on the matter. In case someone is curious, here is the answer:
Thought for 15 seconds
I understand that you're trying to determine whether lowering the "Films in common minimum" to 3% is optimal for you on Criticker. Let me help clarify how this setting works and whether your observations make sense.
Understanding "Films in Common Minimum":
The "Films in common minimum" setting on Criticker determines the minimum percentage of films that another user must have rated in common with you for their ratings to be considered in calculating your Taste Compatibility Index (TCI). The TCI is used to predict how closely another user's tastes align with yours, which in turn affects the Predictive Score Indicator (PSI) for films you haven't seen.
What Happens When You Lower the Minimum Percentage:
Increased Pool of Users: Lowering the minimum percentage allows more users to be included in your TCI calculations, including those who have rated very few films but happen to have high agreement with you on those films.
Potential for Higher Agreement: With more users included, especially those who have rated the same niche or less popular films you've rated, you might find users whose tastes align more closely with yours.
Improved Prediction Metrics: As you've observed, your average difference decreased, "spot-on" predictions increased, and "way-off" predictions decreased. This suggests that the PSIs are aligning more closely with your actual ratings.
Potential Downsides and Biases:
Overfitting: By including users who have only a few films in common with you, the system might overfit to those specific films, which may not generalize well to other films. This means the predictions might be accurate for a small set of films but less reliable overall.
Variability and Instability: Users with fewer ratings can introduce more variability. Since their TCI with you is based on a small number of films, any disagreement on additional films can significantly affect their compatibility score.
Limited Discovery: As "geohawk" mentioned, lowering the minimum percentage might limit the breadth of recommendations. You might miss out on films that are outside your usual preferences but that you might still enjoy.
Addressing "geohawk"'s Response:
Bias and Breadth: "Geohawk" points out that lowering the minimum percentage can introduce bias because you're favoring users who have rated the same specific films as you. This can improve prediction accuracy for those films but might not help with discovering new films.
Trade-Off Between Accuracy and Discovery: There's a balance between getting highly accurate predictions for a narrow set of films and getting broader recommendations that might be less accurate but introduce you to new content.
Dependence on Individual Preferences: The optimal setting can vary depending on your rating habits and the diversity of films you've rated. If you've rated many niche films, lowering the minimum percentage might indeed yield better predictions for you.
Conclusion:
Based on your observations and metrics, it seems that lowering the "Films in common minimum" to 3% has improved the prediction accuracy for you. This suggests that, in your specific case, 3% might be the optimal setting. However, it's essential to be aware of the potential downsides:
Overfitting to Specific Users: Your predictions might be heavily influenced by a small number of users with whom you share niche preferences.
Reduced Variety in Recommendations: You might miss out on broader recommendations that could help you discover new films.
Recommendations:
Monitor Over Time: Continue using the 3% setting and monitor whether the improved accuracy holds as you rate more films.
Experiment with Different Settings: Try other settings between 3% and 25% to see if there's a sweet spot that balances accuracy and discovery.
Consider Your Goals: If your primary goal is accurate predictions for films similar to those you've already rated, the lower percentage might be best. If you're interested in broader recommendations, a higher percentage might be beneficial.
Final Thoughts:
Yes, your findings are valid, and it's possible that 3% is the optimal setting for you given your unique rating history and preferences. "Geohawk" provides a general caution about potential biases and the trade-offs involved, which are essential to consider. Ultimately, the best setting depends on your personal goals and how you weigh prediction accuracy against the desire for broader recommendations.
-------------
Comments are welcome!
Thought for 15 seconds
I understand that you're trying to determine whether lowering the "Films in common minimum" to 3% is optimal for you on Criticker. Let me help clarify how this setting works and whether your observations make sense.
Understanding "Films in Common Minimum":
The "Films in common minimum" setting on Criticker determines the minimum percentage of films that another user must have rated in common with you for their ratings to be considered in calculating your Taste Compatibility Index (TCI). The TCI is used to predict how closely another user's tastes align with yours, which in turn affects the Predictive Score Indicator (PSI) for films you haven't seen.
What Happens When You Lower the Minimum Percentage:
Increased Pool of Users: Lowering the minimum percentage allows more users to be included in your TCI calculations, including those who have rated very few films but happen to have high agreement with you on those films.
Potential for Higher Agreement: With more users included, especially those who have rated the same niche or less popular films you've rated, you might find users whose tastes align more closely with yours.
Improved Prediction Metrics: As you've observed, your average difference decreased, "spot-on" predictions increased, and "way-off" predictions decreased. This suggests that the PSIs are aligning more closely with your actual ratings.
Potential Downsides and Biases:
Overfitting: By including users who have only a few films in common with you, the system might overfit to those specific films, which may not generalize well to other films. This means the predictions might be accurate for a small set of films but less reliable overall.
Variability and Instability: Users with fewer ratings can introduce more variability. Since their TCI with you is based on a small number of films, any disagreement on additional films can significantly affect their compatibility score.
Limited Discovery: As "geohawk" mentioned, lowering the minimum percentage might limit the breadth of recommendations. You might miss out on films that are outside your usual preferences but that you might still enjoy.
Addressing "geohawk"'s Response:
Bias and Breadth: "Geohawk" points out that lowering the minimum percentage can introduce bias because you're favoring users who have rated the same specific films as you. This can improve prediction accuracy for those films but might not help with discovering new films.
Trade-Off Between Accuracy and Discovery: There's a balance between getting highly accurate predictions for a narrow set of films and getting broader recommendations that might be less accurate but introduce you to new content.
Dependence on Individual Preferences: The optimal setting can vary depending on your rating habits and the diversity of films you've rated. If you've rated many niche films, lowering the minimum percentage might indeed yield better predictions for you.
Conclusion:
Based on your observations and metrics, it seems that lowering the "Films in common minimum" to 3% has improved the prediction accuracy for you. This suggests that, in your specific case, 3% might be the optimal setting. However, it's essential to be aware of the potential downsides:
Overfitting to Specific Users: Your predictions might be heavily influenced by a small number of users with whom you share niche preferences.
Reduced Variety in Recommendations: You might miss out on broader recommendations that could help you discover new films.
Recommendations:
Monitor Over Time: Continue using the 3% setting and monitor whether the improved accuracy holds as you rate more films.
Experiment with Different Settings: Try other settings between 3% and 25% to see if there's a sweet spot that balances accuracy and discovery.
Consider Your Goals: If your primary goal is accurate predictions for films similar to those you've already rated, the lower percentage might be best. If you're interested in broader recommendations, a higher percentage might be beneficial.
Final Thoughts:
Yes, your findings are valid, and it's possible that 3% is the optimal setting for you given your unique rating history and preferences. "Geohawk" provides a general caution about potential biases and the trade-offs involved, which are essential to consider. Ultimately, the best setting depends on your personal goals and how you weigh prediction accuracy against the desire for broader recommendations.
-------------
Comments are welcome!