What's The Deal With MMR

Competitive Discussion
Prev 1 2 3 4 21 Next
10/19/2016 05:08 PMPosted by HimeSama
10/19/2016 02:28 PMPosted by Scott Mercer
So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about


Here is my question: How can a mathematical equation be "uncertain".

I know you can't let us into the heads of your devs or reveal proprietary code, but I'm just curious. Not attacking or anything. Perhaps understanding how the system works will help people not get so frustrated when the MMR/SR system is taking a big one on their day.


The principle of "uncertainty" is in reference to measuring a player's skill. Skill fluctuates, that's just a given. You have good days and bad days. Your MMR is never supposed to cement at a fixed value. It's supposed to ebb and flow. Uncertainty just measures how much it ebbs and flows. It's not the calculation that's uncertain, it's the measurement of confidence in your past results shaping the system's current impression of your skill.

Let's say I play a match against you and we're both brand new players. We both start at the seed value of 2500. You beat me. My rating drops to 2450 and yours increases to 2550. Is 2550 an accurate representation of your true potential? Probably not. It's going to require more games -- more data points -- to determine whether that current rating is accurate. If you keep winning and winning and winning, the confidence in whether you are really 2550 is very low. If you keep losing and losing and losing, then you getting to 2550 in the first place may have been a fluke, so the confidence in whether you are really 2550 in that scenario is also very low. If you happen to go 50/50 by playing against other 2550 opponents, then confidence will be a bit higher that that's roughly the range you should be at (depending on how consistent you are), so it makes sense to reduce the volatility a bit and reduce your rating change, since we have sufficient data to conclude that you actually belong there.
As much as I love the post explaining match making something definitely needs improving. http://imgur.com/W3Hor80. This happened at 11:47am PST today and the 6 man said that they waited 6mins. When the average match takes 15 to 20 mins to complete 6 mins is way too short. I get it that it sucks for a 6 man to wait for a match but a match like this doesn't help anyone and is a waste of both teams times.
10/19/2016 05:08 PMPosted by HimeSama
10/19/2016 02:28 PMPosted by Scott Mercer
So then why do points for losses and wins seem so random? Well, the amount of MMR (and SR) you go up or down isn't simply a matter of whether you won or lost, and what was your predicted chance of winning. There's a couple of other things at work. One is the matchmaker's confidence in what your MMR should be. Play a lot of games, it gets more certain. Don't play Overwatch for a while, it gets less certain. You go on a large win or loss streak, it gets less certain. The more certain the matchmaker is about


Here is my question: How can a mathematical equation be "uncertain".

I know you can't let us into the heads of your devs or reveal proprietary code, but I'm just curious. Not attacking or anything. Perhaps understanding how the system works will help people not get so frustrated when the MMR/SR system is taking a big one on their day.

Uncertainty exists even in the very base ELO Rating System, in which it is known as a K value. It mostly exists as a weight to increase changes to your MMR/SR when it is perceived to not reflect your actual level of skill, for example a new account or smurf, or someone taking a break for an extended period of time, or someone streaking wins/losses in their current rating. As more matches are played and the formula seems to be more accurately placing you at a rating where the calculated 50% chance of winning is actually happening, uncertainty value decreases.
So the biggest wrench to the system is those trolls and quitters huh? Because I definitely don't do well 5v6... usually ends in a loss which goes towards ending any winning streaks and greatly contributes to losing streaks...
Solo only queue IS the answer. It creates an equal playing field which is essential for a competitive game, end of discussion. Were not looking to fix any other issue here. We want a ladder representing the skill of a player, thus we have to have every player facing the same environment/stuggle. People that queue with better players than themselves and face solo players are not being given the same challenge as 100% solo queue players. Grouped players have a statistic advantage. There shouldn't be any advantage or disadvantage in a competitive ladder.

When you look at a top 500 player you should feel impressed because they are facing what you're facing and you KNOW that for a fact with a solo only queue. There are many players that have stated that their success past diamond has been largely contributed to those that they play with. When I see a top 500 player, I see a player that was fortunate enough to find the right group, not someone who is objectively deserving of the position. (This isn't true all the time, but there are enough cases of this to imply a problem.)
10/19/2016 05:20 PMPosted by D4G
As much as I love the post explaining match making something definitely needs improving. http://imgur.com/W3Hor80. This happened at 11:47am PST today and the 6 man said that they waited 6mins. When the average match takes 15 to 20 mins to complete 6 mins is way too short. I get it that it sucks for a 6 man to wait for a match but a match like this doesn't help anyone and is a waste of both teams times.


Yeah, that's a situation we'd definitely like to improve. The core problem is that there might not be a fair match for that 6 player group in your screenshot, not just after 6 minutes but after 15, 30, or 60. I know that people say that they're willing to wait longer, but there are also practical limits to that patience. We've been erring more on the side of people at least playing the game instead of sitting in queue. Those 6 player stacks sometimes lose, too.
So what if you like myself consistently get lose and win streaks? pretty much explains why it is so random if the system has no idea what skill I should be...

The system explained here sounds very reasonable, and I feel that when you play with a six stack it's pretty much like this, you win some you lose some and overall it feels rather fair.
But when you start playing with less things start to break down fast, me being primarily a solo player it is just a big mess. There seems to be no consistency to it at all.
10/19/2016 05:48 PMPosted by Scott Mercer
10/19/2016 05:20 PMPosted by D4G
As much as I love the post explaining match making something definitely needs improving. http://imgur.com/W3Hor80. This happened at 11:47am PST today and the 6 man said that they waited 6mins. When the average match takes 15 to 20 mins to complete 6 mins is way too short. I get it that it sucks for a 6 man to wait for a match but a match like this doesn't help anyone and is a waste of both teams times.


Yeah, that's a situation we'd definitely like to improve. The core problem is that there might not be a fair match for that 6 player group in your screenshot, not just after 6 minutes but after 15, 30, or 60. I know that people say that they're willing to wait longer, but there are also practical limits to that patience. We've been erring more on the side of people at least playing the game instead of sitting in queue. Those 6 player stacks sometimes lose, too.


Capping the amount of people you can queue with is one solution. Perhaps you can only queue with a maximum of 1 or 2 people and that would help find a match which suits them more of the time.

There is an argument that at the higher ranks (master and above) this should be enforced to prevent exploitation of the system to some degree.
Haw they claim there is millions of players. If there are millions then 12 players should be a quick match and shouldn't take 3,4 minutes to find a match solo.

The MMR is like MCI math. Look deep in the crystal ball for answers.
10/19/2016 02:28 PMPosted by Scott Mercer
The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.


I'm not sure what you changed regarding this, if anything, from Season One to Season Two but I noticed a vast increase in groups vs solo queue or unequal groupings in Season Two while in Season One it was a very rare occurrence. I actually have a hard time recalling when group matching was uneven at all during Season One.
10/19/2016 05:01 PMPosted by Scott Mercer
...

These things. Seems streaks only serve to undo the system's attempts to remain accurate.


We've actually done some tuning since launch so it takes several more wins or losses in a row to trigger full expansion of the uncertainty due to streaks. It's been tricky to tune. We don't want to have your SR seesawing wildly due to natural variance in wins and losses, but but we do want it to be meaningful enough so if someone's MMR is incorrect (a new account, someone trying to smurf, etc.) we get them to their "proper" MMR quickly so they're playing against the players who they should be playing.

shouldn't the part of uncertainty that starts with no confidence in your rating early take care of that part? if you go say 7-3 in the next 10 games after placements end, the system still isn't confident in your rating and the rank change stays large, compared to going 7-3 over say games 100-110. the winstreak itself should have no bearing on a smurf account because it has to start with placements just like any other, and its early season uncertainty should handle that problem.

thats for the discussion scott, we've been thirsting for it.
10/19/2016 06:50 PMPosted by Sylhiri
10/19/2016 02:28 PMPosted by Scott Mercer
The system does in fact try to place equal sized groups on opposite teams whenever possible. Your report for game 2 where both 3 player groups were on the same team definitely seems like something that shouldn't happen based on the rules we've setup, so I'll look into it further.


I'm not sure what you changed regarding this, if anything, from Season One to Season Two but I noticed a vast increase in groups vs solo queue or unequal groupings in Season Two while in Season One it was a very rare occurrence. I actually have a hard time recalling when group matching was uneven at all during Season One.


It was a plague in Season 1 at the top level with SR70+s grouping with people who were legitimately SR70 but had intentionally dropped to SR30 in order to lower the group average and allow for easy wins and an effective rating boost.
10/19/2016 05:48 PMPosted by Scott Mercer
10/19/2016 05:20 PMPosted by D4G
As much as I love the post explaining match making something definitely needs improving. http://imgur.com/W3Hor80. This happened at 11:47am PST today and the 6 man said that they waited 6mins. When the average match takes 15 to 20 mins to complete 6 mins is way too short. I get it that it sucks for a 6 man to wait for a match but a match like this doesn't help anyone and is a waste of both teams times.


Yeah, that's a situation we'd definitely like to improve. The core problem is that there might not be a fair match for that 6 player group in your screenshot, not just after 6 minutes but after 15, 30, or 60. I know that people say that they're willing to wait longer, but there are also practical limits to that patience. We've been erring more on the side of people at least playing the game instead of sitting in queue. Those 6 player stacks sometimes lose, too.

I'll be frank I would think ~6 minutes between matches ought to, in most situations, be enough to find a relatively good matchup for a 6-stack.

I've had some weird drops recently, including one where we ended up against another 6-stack almost 300 rating average higher than us. (For the gratification of the crowd, we won that match. Playing Bastion, Symmetra, Torbjorn, Mei, Mercy, and Zenyatta. On King of the Hill.)

These don't bug me nearly as much as finding 6-stacks dropped in against 2 3-stacks or a 4-stack + 2-stack; that's simply an unassailable advantage in communication and play familiarity.

There was a lot of questioning vis-a-vis why there isn't a strictly 6v6 competitive mode early on, which has sort of died down as the whinier folks found greener pastures. But it's worth asking why these game modes that would clearly improve QOL for a lot of your players aren't reaching PTR, at least, if not live builds.

Thanks for answering matchmaker questions. By far the most perplexing aspect of this game. Surpassing even Schrodinger's Flashbang.
10/19/2016 07:46 PMPosted by ExcaliburZ

It was a plague in Season 1 at the top level with SR70+s grouping with people who were legitimately SR70 but had intentionally dropped to SR30 in order to lower the group average and allow for easy wins and an effective rating boost.


I'm talking about the consistency of pairing groups with groups in matchmaking. While their was problems (more so at the top level) about people manipulating the SR to get into lower SR matches (somewhat fixed with Season Two), I never saw (that I could remember) a party of 6 go against all solo queue players.

If the match had a group of 2 with 4 solo queue players you can bet that the other team had a group of 2 with 4 solo queue players, in my experience in Season One I never saw a group of solo queue players go up against a party of any kind. I don't even recall odd number of groupings like (2)+(4) vs (6), it was always (2)+(4) vs (2)+(4) or (6) vs (6). Season Two one of my early matches was a group of 6 vs pure solo queue.
10/19/2016 05:01 PMPosted by Scott Mercer
We've actually done some tuning since launch so it takes several more wins or losses in a row to trigger full expansion of the uncertainty due to streaks. It's been tricky to tune. We don't want to have your SR seesawing wildly due to natural variance in wins and losses, but but we do want it to be meaningful enough so if someone's MMR is incorrect (a new account, someone trying to smurf, etc.) we get them to their "proper" MMR quickly so they're playing against the players who they should be playing.


Yeah, makes sense wanting to move people to the correct MMR as quickly as possible, especially if they're at ranks beneath their actual skill and possibly hurting the game experience for others. However, I can think of times when I've lost several games in a row, and while the games were very close and very competitive, I found myself losing quite a bit of SR just because they happened to be consecutive. Those types of streaks are frustrating enough, and the extra drop in SR just feels like salt in the wound much of the time.

Personally, I'd rather deal with the occasional smurf or fresh player who's dominating a few games on their way up, than to feel the added pressure on my SR if a few games in a row happen to be losses, especially if they're close games. Just my opinion though.

Glad to hear it's been adjusted so more wins/losses are required for streaks, because I feel like it would take a solid 8 or 10 games in a row to really indicate someone isn't where they should be.

Thanks for the replies here!
Hi Scott nice post but unfortunately seeing two 3 stacks on the same team vs 6 solo players happens so often that my eyebrows have now raised out of the ceiling.
10/19/2016 08:28 PMPosted by Sciron
Glad to hear it's been adjusted so more wins/losses are required for streaks, because I feel like it would take a solid 8 or 10 games in a row to really indicate someone isn't where they should be.

Thanks for the replies here!


According to my tracking (https://docs.google.com/spreadsheets/d/1TYxqOPL7zBauizxk7js5OH-CCbncW14HNOZkt594jLg/edit?usp=sharing), in Season 1 the SR change really kicked in on the 3rd win/loss in a row. In Season 2 it appears to kick in after the 4th or 5th.
10/19/2016 04:51 PMPosted by Scott Mercer
10/19/2016 03:01 PMPosted by Light
...

Does this mean that as time goes on it will be even more difficult to rise or fall in rank?

Based on what you said, it sounds like the more you play the more certain the system is on where you should be. However, over time I hope to improve. Will the system thinking that I am where I should be have a negative result on me climbing in the future (via lower amount of SR gain from wins)?


Not in the long term. Something to remember is that when uncertainty decreases you also won't lose as much SR when you do lose a match.


So are you saying that certainty is a variable that goes up over the matches you play but decreases if you go on a streak?
10/19/2016 08:35 PMPosted by ExcaliburZ
According to my tracking (https://docs.google.com/spreadsheets/d/1TYxqOPL7zBauizxk7js5OH-CCbncW14HNOZkt594jLg/edit?usp=sharing), in Season 1 the SR change really kicked in on the 3rd win/loss in a row. In Season 2 it appears to kick in after the 4th or 5th.


Oh nice, thanks for sharing this.

I was under the impression that it was after 4 or 5 wins/losses currently, but wasn't sure if Scott was referring to the current game or changes that haven't gone live yet (rereading I think he's probably talking about current game).

I'm glad they moved in the right direction, but I think it needs to be a much more significant streak before the big changes kick in.
I would just like to say thank you for your transparency on this. A lot of people have been quite frustrated for a while, this will hopefully alleviate some stress and concern within the community.

Keep it up guys! You are doing great work with this!

I would also like to know, do you factor in last minute hero swaps to keep overtime ticking in your individual performance?
For example, if I pick D.Va on the last point to try and stall for time, I get 2 deaths, deal basically no damage, and don't get an ult.
If we compared that to the historical data for D.Va players, it would be very poor. Do you exclude swaps like this?

Join the Conversation

Return to Forum