What's The Deal With MMR

Competitive Discussion
Prev 1 2 3 21 Next
A bit off topic, but maybe Scott Mercer could explain this specific case?
So i had a super weird match last night.
Almost equal sr on both ends, differing by about 30 i think? Since it was in the middle of the night with not many ppl playing, it mixed high platin/low diamond and 3 top500 players.
The average sr on our team was about 2930, the enemys was 2960, something like that.
Our team was all solo players, 1 top 500 player.
Enemy team had the other 2 top500 players grouped together.
So our team shouldve been the underdog right?
Yet after winning 3:1 on lijiang tower, all on our team ended up gaining exactly 2 sr.
How?
The more I think about it the more I think the blue post today is a band aid to stop the player base bleeding out of OW.

Bet it will be quite a while before we see another blue post or a game fixing update...
10/19/2016 02:28 PMPosted by Scott Mercer
Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.


This is the only part I really have a problem with. You say that "you shouldn't lose much SR", however the loss still effects your win streak, or still adds to your loss streak - so effectively you are losing a significant amount of SR.

This is a big problem when the SR average is so different, we have had cases where the other team has 500 greater SR average with top 500 players, and we have no chance of winning.

Have you considered maybe, if the group average difference of the other team is > 250, then the loss doesn't break your win streak, and doesn't contribute to your loss streak?
The problem I have with this whole thing is that the game supposedly has millions of players. At least 20% of my games feature premades of some sort against PUGs.

This is either on my team or the enemies. I've even had a few where I queue into a 6 man team when I solo queue. It shouldn't happen, ever. It should be impossible.

Situations like these arise far too often for me to have any confidence in the matchmakers ability, not to mention that the 95% of my games are either an easy win, or a stomp. I rarely play close/decent games and that's a shame.
10/19/2016 02:28 PMPosted by Scott Mercer
Thanks for the well written post on your matchmaking experiences, I'll try to explain some of what's going on.

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.

We do need to do a better job of not placing players into "unwinnable" matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you're a player closer to median skill rating and you're not in a group), but I've definitely seen logs of matches where that's really not the case and my eyebrows raise.

The unfortunate truth is that there is not always a "perfect" match for you, especially at very high (and very low!) skill ratings where there's fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak... it can get tough. We've tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we're always looking to improve and tune better.

Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.

For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)

The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here's where things get interesting. This "massive" advantage actually differs based upon the skill rating of the group members. Based upon the data we've seen groups of low to mid SR players don't see that much improvement to their win %. Higher SR players do see more notable improvements, but it's not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there's a perception of a large advantage for groups. That's one of the reasons why we explicitly try to match similar sized groups together.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we've done a lot of work to this system based on the community's feedback. In fact, I've seen some people indicate that they don't think we're doing this anymore. We still are. While it's a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.

So take all that into account, the SR gain/loss after any single match can be a bit more "noisy" that it seems it should, but we're asking it to look at a lot of different factors to do the best job it can creating fair matches for you.

As an aside to all of this...

"Fair" matches doesn't always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It's not always the matchmaker's, your's, or your team's(!!!) fault that you got stomped.


The biggest problem I have with the system is sometimes the matches just don't feel worth playing. In matches where the system thinks I should win and predicts I have a high chance of winning, I have very little to gain, but very much to lose. So if there's a leaver or troll on my team, or we just don't do well together, I lose way too much SR because the system expected me to win. And when we win, the game just feels like a waste of time because I hardly gain any SR and I worked hard for basically nothing.
10/19/2016 02:28 PMPosted by Scott Mercer
Thanks for the well written post on your matchmaking experiences, I'll try to explain some of what's going on.

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.

We do need to do a better job of not placing players into "unwinnable" matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you're a player closer to median skill rating and you're not in a group), but I've definitely seen logs of matches where that's really not the case and my eyebrows raise.

The unfortunate truth is that there is not always a "perfect" match for you, especially at very high (and very low!) skill ratings where there's fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak... it can get tough. We've tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we're always looking to improve and tune better.

Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.

For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)

The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here's where things get interesting. This "massive" advantage actually differs based upon the skill rating of the group members. Based upon the data we've seen groups of low to mid SR players don't see that much improvement to their win %. Higher SR players do see more notable improvements, but it's not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there's a perception of a large advantage for groups. That's one of the reasons why we explicitly try to match similar sized groups together.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we've done a lot of work to this system based on the community's feedback. In fact, I've seen some people indicate that they don't think we're doing this anymore. We still are. While it's a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.

So take all that into account, the SR gain/loss after any single match can be a bit more "noisy" that it seems it should, but we're asking it to look at a lot of different factors to do the best job it can creating fair matches for you.

As an aside to all of this...

"Fair" matches doesn't always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It's not always the matchmaker's, your's, or your team's(!!!) fault that you got stomped.


Why do winstreaks/losestreaks seem to matter so much? What indicator does that have on a player's skill? And why does it seem like sometimes you can just end up on a series of winstreaks and losestreaks as if the game is intentionally putting you on those?
10/19/2016 02:28 PMPosted by Scott Mercer
Thanks for the well written post on your matchmaking experiences, I'll try to explain some of what's going on.

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.

We do need to do a better job of not placing players into "unwinnable" matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you're a player closer to median skill rating and you're not in a group), but I've definitely seen logs of matches where that's really not the case and my eyebrows raise.

The unfortunate truth is that there is not always a "perfect" match for you, especially at very high (and very low!) skill ratings where there's fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak... it can get tough. We've tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we're always looking to improve and tune better.

Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.

For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)

The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here's where things get interesting. This "massive" advantage actually differs based upon the skill rating of the group members. Based upon the data we've seen groups of low to mid SR players don't see that much improvement to their win %. Higher SR players do see more notable improvements, but it's not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there's a perception of a large advantage for groups. That's one of the reasons why we explicitly try to match similar sized groups together.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we've done a lot of work to this system based on the community's feedback. In fact, I've seen some people indicate that they don't think we're doing this anymore. We still are. While it's a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.

So take all that into account, the SR gain/loss after any single match can be a bit more "noisy" that it seems it should, but we're asking it to look at a lot of different factors to do the best job it can creating fair matches for you.

As an aside to all of this...

"Fair" matches doesn't always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It's not always the matchmaker's, your's, or your team's(!!!) fault that you got stomped.


In other words.. It's a !@#$ed up system, we know it is but tough %^-*. Deal with it.
thanks a load for the most transparent explanation of skill rating to date. seriously. thank you.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

[/quote]
"large?" the win/loss streak effect kicks in well before the streak is large at all. we win or lose a few games in a row all the time. it doesn't mean our skill changed all of a sudden, it means we're playing great or terrible right now, or had luck with the matchmaker (or didn't). and yet we start seeing very noticeable effects after about 3 games. this is a problem, man. this throws noise into the data that it doesn't need.

the other two inputs into certainty make plenty of sense but it's silly for the game to think because i've won 4 in a row, maybe i'm not the same player i've been for well over 200 games. or because i've lost 4, i might be a much worse player than i was yesterday on that +4 streak. someone's skill changes, it's going to reflect in their winrate. let it imo. trying to hurry that process along is just making a mess out of the median ranks.
The ethic of this system still seems horribly off.

Simply put, make all win and loss gains flat across all team mates.

Second, for the love of everything, make your formula public.
10/19/2016 04:22 PMPosted by Brewmeister
10/19/2016 02:28 PMPosted by Scott Mercer
Thanks for the well written post on your matchmaking experiences, I'll try to explain some of what's going on.

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.

We do need to do a better job of not placing players into "unwinnable" matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you're a player closer to median skill rating and you're not in a group), but I've definitely seen logs of matches where that's really not the case and my eyebrows raise.

The unfortunate truth is that there is not always a "perfect" match for you, especially at very high (and very low!) skill ratings where there's fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak... it can get tough. We've tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we're always looking to improve and tune better.

Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.

For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)

The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here's where things get interesting. This "massive" advantage actually differs based upon the skill rating of the group members. Based upon the data we've seen groups of low to mid SR players don't see that much improvement to their win %. Higher SR players do see more notable improvements, but it's not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there's a perception of a large advantage for groups. That's one of the reasons why we explicitly try to match similar sized groups together.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we've done a lot of work to this system based on the community's feedback. In fact, I've seen some people indicate that they don't think we're doing this anymore. We still are. While it's a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.

So take all that into account, the SR gain/loss after any single match can be a bit more "noisy" that it seems it should, but we're asking it to look at a lot of different factors to do the best job it can creating fair matches for you.

As an aside to all of this...

"Fair" matches doesn't always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It's not always the matchmaker's, your's, or your team's(!!!) fault that you got stomped.


In other words.. It's a !@#$ed up system, we know it is but tough %^-*. Deal with it.

This is why devs never respond on here.

To the point I'm glad to hear they are still working on the system nothing worse than winning a hard fought game and gaining 2 points, and if you lose that game taking a gigantic loss.
Does this mean that as time goes on it will be even more difficult to rise or fall in rank?

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?


And what happens in the end is that the smart system forces you to what it thinks you should be but because you lose more points than you win, then it is not enough to improve, you actually have to become extremely good (or extremely lucky) )to be able to actually climb again.

And... Despite what it says that it tries to put you in 50/50 matches, it really seems like after winning a few you are put in a match that you have barely any chance to win.

I don't know how to solve the problem because, I might be really good with one hero and really bad with another but in the end, it is judging me and I could end up playing the game of my life and still losing one after another and sinking more and more into the "I-don't-give-a-damn-anymore" region.
10/19/2016 03:16 PMPosted by Nessuno
The effect of win/loss streaks seems to be the biggest culprit in the noise, and adds a lot of unnecessary randomness.

Whether or not your wins are strung together will have a huge impact on your "season high", which determines both your end of season reward and your displayed tier level.


10/19/2016 04:25 PMPosted by stephofni
the other two inputs into certainty make plenty of sense but it's silly for the game to think because i've won 4 in a row, maybe i'm not the same player i've been for well over 200 games. or because i've lost 4, i might be a much worse player than i was yesterday on that +4 streak. someone's skill changes, it's going to reflect in their winrate. let it imo. trying to hurry that process along is just making a mess out of the median ranks.


These things. Seems streaks only serve to undo the system's attempts to remain accurate.
10/19/2016 02:28 PMPosted by Scott Mercer
For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)

[/quote]

Why "Hint"? What would be so bad about telling us what the actual model being used is?
Don't worry about MMR. Just play your best and know you can win every match you join if you try hard enough. Lazy players wont get far...
10/19/2016 03:01 PMPosted by Light
10/19/2016 02:28 PMPosted by Scott Mercer
One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.


Does this mean that as time goes on it will be even more difficult to rise or fall in rank?

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?


Not in the long term. Something to remember is that when uncertainty decreases you also won't lose as much SR when you do lose a match.
10/19/2016 04:51 PMPosted by Scott Mercer
10/19/2016 03:01 PMPosted by Light
...

Does this mean that as time goes on it will be even more difficult to rise or fall in rank?

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?


Not in the long term. Something to remember is that when uncertainty decreases you also won't lose as much SR when you do lose a match.


None of this makes competitive games more fun to play.

Your format is great! Swapping sides, pick one hero per team only. But the ranking system itself doesn't translate into more fun games, for whatever reason.
10/19/2016 04:54 PMPosted by KelRiever
10/19/2016 04:51 PMPosted by Scott Mercer
...

Not in the long term. Something to remember is that when uncertainty decreases you also won't lose as much SR when you do lose a match.


None of this makes competitive games more fun to play.

Your format is great! Swapping sides, pick one hero per team only. But the ranking system itself doesn't translate into more fun games, for whatever reason.
It's not supposed to make it more fun, It's supposed to make it more competitive.
10/19/2016 04:42 PMPosted by Sciron
10/19/2016 03:16 PMPosted by Nessuno
The effect of win/loss streaks seems to be the biggest culprit in the noise, and adds a lot of unnecessary randomness.

Whether or not your wins are strung together will have a huge impact on your "season high", which determines both your end of season reward and your displayed tier level.


10/19/2016 04:25 PMPosted by stephofni
the other two inputs into certainty make plenty of sense but it's silly for the game to think because i've won 4 in a row, maybe i'm not the same player i've been for well over 200 games. or because i've lost 4, i might be a much worse player than i was yesterday on that +4 streak. someone's skill changes, it's going to reflect in their winrate. let it imo. trying to hurry that process along is just making a mess out of the median ranks.


These things. Seems streaks only serve to undo the system's attempts to remain accurate.


We've actually done some tuning since launch so it takes several more wins or losses in a row to trigger full expansion of the uncertainty due to streaks. It's been tricky to tune. We don't want to have your SR seesawing wildly due to natural variance in wins and losses, but but we do want it to be meaningful enough so if someone's MMR is incorrect (a new account, someone trying to smurf, etc.) we get them to their "proper" MMR quickly so they're playing against the players who they should be playing.
10/19/2016 05:01 PMPosted by Voidz
10/19/2016 04:54 PMPosted by KelRiever
...

None of this makes competitive games more fun to play.

Your format is great! Swapping sides, pick one hero per team only. But the ranking system itself doesn't translate into more fun games, for whatever reason.
It's not supposed to make it more fun, It's supposed to make it more competitive.


First, it doesn't do that either.

Second, so you are saying competition isn't fun. Got it.
10/19/2016 02:28 PMPosted by Scott Mercer
So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about


Here is my question: How can a mathematical equation be "uncertain".

I know you can't let us into the heads of your devs or reveal proprietary code, but I'm just curious. Not attacking or anything. Perhaps understanding how the system works will help people not get so frustrated when the MMR/SR system is taking a big one on their day.

Join the Conversation

Return to Forum