What's The Deal With MMR

Competitive Discussion
1 2 3 21 Next
Highly Rated
So this has probably been said a thousand times in more ranty fashion, but I'm legitimately curious if there is any reasoning behind the gain and loss of MMR points and if Blizzard has any intention of improving the system.

At present, there doesn't appear to be much if any logic to how many points you gain or lose in competitive. This is exacerbated by the fact that the matchmaking system occasionally throws you into virtually unwinnable games.

Example from my play this morning:

First game (pretty fair match)
Team 1 - Team rank 3400, 3 pugs and 1 3 premade, personal rating 3505
Team 2 - Team rank 3402, 3 pugs and 1 3 premade
Won - 19 points gained

Second game (!@#$ing what the actual %^-* blizzard)
Team 1 - Team rank 3550, 6 pugs, personal rating 3524
Team 2 - Team rank 3600, 2 x 3 premade
Lost (duh) - 27 points lost

Third game (pretty fair match)
Team 1 - Team rank 3497, 6 pugs, personal rating 3497
Team 2 - Team rank 3505, 6 pugs
Won - 20 points gained

Fourth game (MMR gods show me favor)
Team 1 - Team rank 3557, 3 pugs and 1 3 premade, personal rating 3517
Team 2 - Team rank 3496, 6 pugs
Won - 23 points gained

I get that this doesn't seem like a dramatic swing, but think about this. If the MMR system is working as intended you would expect to gain an average amount of points for a fair match victory, a large amount of points for an underdog victory, and a small amount of points for an expected victory. And we would also expect the opposite for losses; average loss for a fair match, lower loss for an underdog loss (expected loss), and a large loss for a match you lose despite an expected victory.

The reality is that the system does not appear to work like that whatsoever.

As far as I can tell, these are the biggest issues currently facing competitive MMR:

Matchmaking Does Not Weight Premades Correctly
Simply put, matchmaking isn't taking premades into account the way it should. A 3 man premade all in voice chat and cooperating with each other is a MASSIVE advantage in this game. Placing 2 x 3 premades against 6 pugs should never happen, period. If pugs have to play against a team to keep matchmaking times reasonable, at least match in higher ranked pugs so the team with the premade advantage doesn't also have MMR advantage. Additionally, victories against premades should award larger point values and losses against premades should result in fewer points lost.

Matchmaking Tends To Match Premades Together And Pugs Together
This feeds into the first point, but matchmaking appears to heavily favor putting premades with other premades and pugs with other pugs. It is far more common to see a game with 2 or more premades on one team and 6 pugs on the other. This is particularly frustrating because if a game is 6 pugs (team rank 3400) versus 2 x 3 premades (team rank 3400) then it obviously could have matched that game as 3 pugs + 1 3 man (team rank 3400) versus 3 pugs + 1 3 man (team rank 3400). It is absolutely baffling that lopsided premade matches like these happen and shows that the matchmaking system either isn't taking premades into account at all or is assigning far to small a value to premades.

Points For Losses And Victories Seem Random
The points assigned for a loss or victory do not seem to take into account MMR of either team or premades on either team. In other words, MMR doesn't seem to work period. If the point of the MMR system is to assign a score based on what tier you should be playing in then it by definition needs to weight victories and losses based on your opponents. In it's current form it simply doesn't.
This is a really good post. In my experiences with solos vs stacks, the solos are rated higher (they're generally supposed to be, since that's intended to be a handicap for those situations). In your example, they aren't. On top of that, your rating change was greater than expected, both in game 2 and game 4.

We know that individual performance factors into rating change, but we don't know by how much. Could that really explain the delta?
Thanks for the well written post on your matchmaking experiences, I'll try to explain some of what's going on.

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.

We do need to do a better job of not placing players into "unwinnable" matches. When the matchmaker creates a match, it determines the % chance for each team to win based on the match it made. The VAST majority of matches are usually near to 50% (especially if you're a player closer to median skill rating and you're not in a group), but I've definitely seen logs of matches where that's really not the case and my eyebrows raise.

The unfortunate truth is that there is not always a "perfect" match for you, especially at very high (and very low!) skill ratings where there's fewer players of similar skill. Then you throw in the desire to match groups vs. groups, with everybody having low latency, and doing ALL of this as fast as possible even though it’s the 3AM offpeak... it can get tough. We've tried different tunings with regards to wait times, and the improvements were unfortunately modest as we increased the time to wait. Still, this is an area we're always looking to improve and tune better.

Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.

For matchmaking groups, there's actually two separate issues that we try to solve. The first issue is "How do we handle groups formed of players with different MMRs?". With season 2 we prevented players of REALLY disparate Skill Rating from grouping, but there's still some variance we need to handle. Over time we've tested different models to try and see what's best and are now using what tested most accurately. (Hint: it's not simply averaging the MMRs)

The other issue is how do we model the synergistic effects of players being together in a group. As you noted, they have access to voice chat. Now here's where things get interesting. This "massive" advantage actually differs based upon the skill rating of the group members. Based upon the data we've seen groups of low to mid SR players don't see that much improvement to their win %. Higher SR players do see more notable improvements, but it's not as huge as you might think. Still, we do take this into account when we predict the win% for each team. Regardless of how the data looks, we do know there's a perception of a large advantage for groups. That's one of the reasons why we explicitly try to match similar sized groups together.

So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

As a minor factor, we also do evaluate how well you played the heroes you used in a match. The comparison is largely based on historical data of people playing a specific hero (not medals, not pure damage done), and we've done a lot of work to this system based on the community's feedback. In fact, I've seen some people indicate that they don't think we're doing this anymore. We still are. While it's a minor factor compared to wins/losses (The best way to increase your SR is still to play together and win as a team!), doing so does help us determine your skill more accurately and faster.

So take all that into account, the SR gain/loss after any single match can be a bit more "noisy" that it seems it should, but we're asking it to look at a lot of different factors to do the best job it can creating fair matches for you.

As an aside to all of this...

"Fair" matches doesn't always mean that every Ilios match goes 3-2 and 100-99 on the final point, or each team gets the payload to the end in overtime on Dorado, etc. Sometimes when two evenly matches teams play, the result can be one-sided. It just means that at that single moment in time the enemy team played better. It's not always the matchmaker's, your's, or your team's(!!!) fault that you got stomped.
Highly Rated
Thanks for the post, Scott. I completely agree with the last paragraph and I am so pleased to see you write it.
Sounds good sir.
10/19/2016 02:46 PMPosted by Voidz
Thanks for the post, Jeff.


Tyranny?
Thanks for the reply, Scott.

What we learned here is that:

- Inactivity and streakiness increase uncertainty, resulting in slightly greater rating change (when does it start increasing? 1 day? 1 week?), while additional games reduce uncertainty
- A stack's average SR does not necessarily reflect how they're being matched (there's probably a bias toward the highest player? that's what they learned in Dota...)
- Stacks have a negligible win rate over comparable solo players at low skill, and a minor-to-moderate win rate over comparable solo players at high skill. However, a corresponding handicap is applied if they ever have to play against solos or smaller stacks.
Hey Scott, thanks for the post. While you're here, can you mosey on over to that massive thread begging for the "avoid this player" feature to be re-implemented? Thanks!
Sticky this thread please. It should stop all the people complaining they lose more SR than they gain.
Yea had a few matches with wonky groups that made no sense 3 and 2 vs all pugs for example.
10/19/2016 02:58 PMPosted by kysen
Sticky this thread please. It should stop all the people complaining they lose more SR than they gain.


We shall see.
10/19/2016 02:28 PMPosted by Scott Mercer

The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.


Scott, I'm going to go ahead and say this is absolutely a bug on 3+3 vs 6 solo q. I've seen this A LOT myself. It really stands out as incorrect, like the OP says. In fact, I can't remember the last time (if ever) I played a 3 + 1x3 versus another of the same. In that situation it always ends up 3 +3 vs 6 solo q. (I've seen this watching streamers as well, and it often elicits a comment then too.)
10/19/2016 02:28 PMPosted by Scott Mercer
One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about your MMR, the less your MMR will change in either direction based on a win or loss.

[/quote]

Does this mean that as time goes on it will be even more difficult to rise or fall in rank?

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?
Seems super complicated. I wish we wouldn't get penalized so much for some random who leaves and never comes back. It hurts....so much.
10/19/2016 03:01 PMPosted by Light
Does this mean that as time goes on it will be even more difficult to rise or fall in rank?

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?


As you improve, the better you begin to play. Your individual hero capabilities go up vs the global average + you start winning more matches. These 2 factors increase the uncertainty in determining your SR. It begins to see that you actually don't belong there and will begin to give you more on gains/losses because it's back to trying to determine your true skill rating.
Are you working on reducing volatility of the system? Right now it seems that when you are on a winning streak the system puts you in a group that has average SR much higher than you are, so if you win you get even more. I think I saw in Ster's stream (17 win streak) that he was getting 100 SR per win at the end.
Similarly when you are losing you a losing BIG.
Over my last 200 games or so I was stable between 2200-2300 and recently in 2 days I lost 400 SR. On the second day I lost 220 SR after losing only 6 games (37 per loss on average).
400 SR it's quite a lot and it feel bad, especially considering the fact that I didn't get to that SR on a random wining streak (I was stable there for 200 games).
Today I won 5 out of 7 (3 net) and went up only by 65 SR (22 per win on average).
The effect of win/loss streaks seems to be the biggest culprit in the noise, and adds a lot of unnecessary randomness.

Whether or not your wins are strung together will have a huge impact on your "season high", which determines both your end of season reward and your displayed tier level.
10/19/2016 02:28 PMPosted by Scott Mercer
Fortunately, when we do put you in a match that we know isn't a 50/50, we adjust your SR gain or loss based on your calculated change of winning. So if you did get placed into a match with only a 20% chance to win and then you lose, you shouldn't lose much SR.

Shouldn't the underdog get some SR for a draw in those situations?
10/19/2016 02:28 PMPosted by Scott Mercer
Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.

I just wanted to come into the thread and confirm that this scenario has happened to me multiple times too. It usually seems to be when I play between 3-5pm or +10pm onwards!

Hope it helps :)
Wait, the 2-3 groups thing isn't working as intended?

Almost every other game I'm against a team of groups 2-3 with my team being all Soloque.

Join the Conversation

Return to Forum