Reroll Experiment (With Data!) Part II

Competitive Discussion
TLDR: SR can vary by 1000. SR being off by 500 is normal. Two accounts do not converge in 100+ games.

Part I of the experiment is here: https://us.battle.net/forums/en/overwatch/topic/20753458074

The conclusion of that experiment was that the fastest way to get a bronze/silver account to gold (assuming you have the ability to survive in gold) was by rerolling. The biggest objection to that conclusion was that I hadn’t played enough games, and the accounts would converge with more games. When asked, people said 100 games on each account would be enough. So I played more games on both accounts, continuing to main my robot Zenyatta, but switching when necessary.

Annnnd the result didn’t change. I almost, but didn’t quite get to gold on my silver account before dropping back. I almost, but didn’t quite get to plat on my gold account before dropping down. They are currently 476 apart. The max on my gold account minus the min on my silver account, for season 5 is 916 SR. This will tighten up a bit in future seasons because SR will not be artificially low at the beginning of the season. Regardless, an SR being off almost 1000 appears possible. Both accounts have win rates between 50 and 60% for two seasons, and both are very slowly climbing.

The data:
Main (Kaawumba, https://playoverwatch.com/en-us/career/pc/us/Kaawumba-1133)
Season 4 final SR: 1684
Season 4 high SR: 1730
Record, season 4, including placements: 37-25-3 (59.7%)
Season 5: Placements, 7-3-0, 1582 SR initial placement
Record, season 5, 50 games only, including placements: 27-21-2 (56.3%)
Record, season 5, including placements: 64-59-6 (52.0%)
1844 Current SR, 1972 season high SR
Elims per death: 2.64
Fire per 10 min: 1:19
Healing per 10 min: 5,824
Damage per 10 min: 4,940
Record, seasons 4 & 5: 101-84-9 (54.6%)

Alt (Kaaldrin, https://playoverwatch.com/en-us/career/pc/us/Kaaldrin-1481)
Season 4 final SR: 2283
Season 4 high SR: 2377
Record, season 4, including placements: 24-24-2 (50.0%)
Season 5: Placements, 8-2-0, 2255 initial placement
Record, season 5, 50 games only (and entire season), including placements: 27-22-1 (55.1%)
2320 current SR, 2498 season high SR
Elims per death: 2.09
Fire per 10 min: 0:39
Healing per 10: 5,700
Damage per 10: 3,992
Record, seasons 4&5: 51-46-3 (52.6%)

So, where do I belong? I certainly don’t belong in silver, but it is taking hundreds of games to get out. Perhaps I really belong in plat, but I need a hundred more games to get there. There’s no evidence I belong in diamond plus, but who knows how long I can maintain this slightly over 50% win rate? Of course, at some point you will just conclude that I’m getting better, and that explains why I am steadily climbing with a slightly positive win rate. However, this does not explain why my climb rate in silver isn’t much faster than my climb rate in gold. My conclusions about why this happens haven’t really changed, because my data hasn’t really changed, so I’ll just quote them here, with minor tweaks.

So, why does this happen? Here are some possibilities, in order of likelihood:

Each individual only has a small influence on each game.

Even though having your placement wrong by 800 points seems like a lot, it may not be enough to carry. So, most of the games are determined by factors out of an individual player's control. This is commonly called ELO hell. Certainly a pro could get out much faster, so we know that a 1500-2000 point difference is enough to carry and escape. Perhaps if I chose a different main I could have a greater influence on each game.

The skills required to succeed at bronze are different that the skills required to succeed at gold.

I would say that my mechanics are silver, but my communication skills are gold/plat. So even though I can survive just fine in gold with my communication skills, those do me much less good in silver. This is also a factor in ELO hell.

If I played more games, my accounts would converge a lot faster than hundreds to thousands of games that I'm expecting. I've been lucky/unlucky and that will correct.

This one has been disproved with the extra data.

Blizzard is forcing a 50% win rate by putting potatoes on my team whenever I do well.

No, just no. See also https://us.battle.net/forums/en/overwatch/topic/20753625906#post-13

Possible solutions:

Soft reset every season

A soft reset is one in which the player's rank is preserved, but they gain and lose many more points per match for the beginning of the season. This wouldn't work, at least for me, because my win rate at silver is so close to 50%.

Hard reset every season

This would fix the problem that I am describing, but it would cause a new, much worse problem in which every season would be dominated (much more than now) by people that are placed incorrectly. Ten matches are no where near enough to place people accurately. Essentially, every season would be a massive sort with many, many poor quality matches.

Optional (free) reset

This would fix the problem that I am describing, and would have the lesser problem in which people are thrown out of their proper rank (by choice and luck) and they would screw up other people's games until they go back to their correct rank. This should be limited to perhaps one reset per 100 levels, and can only be done at the beginning of the season. It should be a total wipe, so the player is required to grind out 25 levels of non-competitive before they are allowed back in. I like this solution the best, but I still find it unsatisfactory because it feels more like rerolling (with randomness) than placing someone correctly with a proper algorithm.

Personal stats are weighted more heavily.

This would not really fix the problem. I'd expect motion in the ranks to still be very slow, and it would cause broken behavior is which people care more about the performance stats than actually winning the game. It also makes the rating system even more opaque which is a problem all on its own.

I don't really like any of these solutions. What I'd like is a system that responds fairly quickly to changes in ability, but does not move players if their ability has not changed. The more I think about it, the harder the problem appears. Post your ideas below.
I post from my alt, here, for ID verification purposes.
Thanks for all the data!

I, personally, am in favor of an MMR reset, for plat and below, so that the matchmaker will be forced to do hard swings during placements. Your SR will still be the same but your personal contributions will determine your final placement more heavily.

So instead of always being placed lower than your previous season you might actually be placed above it!
I haven't seen this to be the case in my experience. Perhaps lower ranked players are more liquid but if you have a consistent playstyle a gap of >500 sr should not be the case. It seems like you're implicitly blaming luck - is that your intent?

For me, it has been very noticeable when I reach a rating that I am not prepared for. I get to my season high and get a couple really good teams in a row and before I know it I'm getting wrecked - this becomes noticeable around 100-150 rating outside my comfort zone. The best example for me is 3.3k last season. I was stuck at 3150-3350 for the entire season and whenever I started getting to the upper part of that range higher skilled players would destroy me. However, near the end of the season I became used to playing against these player. After the season ended I took some time off of comp to practice and this season I went up a tier. I hope to do the same next season.

This has been my experience season pretty much season 2. However, I only soloqueue - if you queue with others I have no doubt you'll have a different experience.
Don't you play with a mindset of I lose/win because my team?

That directly correlates to your performance having no impact/inconsistent on every single one of your matches. Thus no matter what tier your get placed in (up to Master, because there will always be !@#$ty players up to that rank), the final outcome of your matches is your team and how well they are able to perform.

When you and this community finally realize how to have an impact on your matches, that's when you'll see where you belong.

Keep playing the way you guys do though, because your skill is currently anywhere from Bronze <-> Master because you don't make impactful plays or even play consistently.
Actually it might be easy to "prove" why you didn't converge yet. on your main account, you seem to be playing way more than the 50 games, ie you play beyond the games where you get bonus, while on your alt account, you only play 50 games, giving you the bonus and nothing else. You also had less stats on higher level, which essentially proves you're doing less than compared to lower rating (which makes sense)

Might be an old record, but I don't think 50 games during the bonus every season is gonna help a lot in trying to determine where you are. You're measuring an account with low games vs a settled account afterall.

Just a theory though.

edit @above - harsh yo.
I don't suppose you ran this experiment blind?
I have an alt account within 200 SR of my main. I also have 2 friends with alt accounts being either same or higher than their main. But all within 300 SR of each other.

I think the only valid reason out of your list is "skill required to succeed at different tiers is vastly different". It makes sense, a very good diamond shotcalling healer main is not going to win games in bronze where its just random death matches and no one goes for objective......they would need to swap to dps and kill things in order to get out of bronze...

MMR reset will make game quality drop a lot.... It would take hundreds of games across entire community to get that bell curve again... I mean I don't mind it because I know where I belong I will get back there easily, I just have to go through a lot of bad games I won't enjoy.

Resetting is not the answer my man, developing multiple sets of skills for different skill tier is the right answer.
...

I think the only valid reason out of your list is "skill required to succeed at different tiers is vastly different". It makes sense, a very good diamond shotcalling healer main is not going to win games in bronze where its just random death matches and no one goes for objective......they would need to swap to dps and kill things in order to get out of bronze...

...

Resetting is not the answer my man, developing multiple sets of skills for different skill tier is the right answer.


This is bad. You don't want people to have to second guess the competency of their teammates and then also learn tactics that are bad in higher level play just to get out of lower level play. That's just bad all around and actually punishes those who exhibit the right mindset and skill set for higher level competitive play. You end up with a rein finding that the only way to win is to dps, but then the same rein finds that one rank higher, shielding is better. It breeds a lack of trust and is generally just a horrible idea. Furthermore, it means that those who conveniently place high enough in the first place don't have to put up with the multiple skill set bull!@#$ while those who actually improve over time do.

The worst part is that you are correct that this is what is required now. I act in the manner that you describe and it works, but that's not how it should work.
08/25/2017 07:25 AMPosted by Mukarramma
I haven't seen this to be the case in my experience. Perhaps lower ranked players are more liquid but if you have a consistent playstyle a gap of >500 sr should not be the case. It seems like you're implicitly blaming luck - is that your intent?

For me, it has been very noticeable when I reach a rating that I am not prepared for. I get to my season high and get a couple really good teams in a row and before I know it I'm getting wrecked - this becomes noticeable around 100-150 rating outside my comfort zone. The best example for me is 3.3k last season. I was stuck at 3150-3350 for the entire season and whenever I started getting to the upper part of that range higher skilled players would destroy me. However, near the end of the season I became used to playing against these player. After the season ended I took some time off of comp to practice and this season I went up a tier. I hope to do the same next season.

This has been my experience season pretty much season 2. However, I only soloqueue - if you queue with others I have no doubt you'll have a different experience.


08/25/2017 07:37 AMPosted by Stebung
I have an alt account within 200 SR of my main. I also have 2 friends with alt accounts being either same or higher than their main. But all within 300 SR of each other.

I think the only valid reason out of your list is "skill required to succeed at different tiers is vastly different". It makes sense, a very good diamond shotcalling healer main is not going to win games in bronze where its just random death matches and no one goes for objective......they would need to swap to dps and kill things in order to get out of bronze...

MMR reset will make game quality drop a lot.... It would take hundreds of games across entire community to get that bell curve again... I mean I don't mind it because I know where I belong I will get back there easily, I just have to go through a lot of bad games I won't enjoy.

Resetting is not the answer my man, developing multiple sets of skills for different skill tier is the right answer.


I have a suspicion that teammate variance is significantly lower in diamond and above than it is in gold and below. Some players are in silver because they a gold level players when they are sober, and bronze level players when they are drunk. Or because they tilt easy. Or they like to switch heroes every game and are much better at some heroes than others. There are more throwers and smurfs in the lower ranks.

It would not be surprising to me if SR accuracy goes up as you go up the ranks. I don't have the means to test it myself, but I'm willing to believe your reports.
08/25/2017 07:26 AMPosted by Takoyakisan
Don't you play with a mindset of I lose/win because my team?

That directly correlates to your performance having no impact/inconsistent on every single one of your matches. Thus no matter what tier your get placed in (up to Master, because there will always be !@#$ty players up to that rank), the final outcome of your matches is your team and how well they are able to perform.

When you and this community finally realize how to have an impact on your matches, that's when you'll see where you belong.

Keep playing the way you guys do though, because your skill is currently anywhere from Bronze <-> Master because you don't make impactful plays or even play consistently.


I'm well aware that the only consistent factor in my games is me, and if I was better my rank would be higher (in both accounts). That isn't really what the post is about. The post is about the reliability (or lack thereof) of SR as a measure of player ability. I consider the game to have the similar mechanics to poker. Better players will win more over the long haul, but in the short-term your cards (other players) can turn against you or save you from your mistakes.
08/25/2017 07:28 AMPosted by Tankz
Actually it might be easy to "prove" why you didn't converge yet. on your main account, you seem to be playing way more than the 50 games, ie you play beyond the games where you get bonus, while on your alt account, you only play 50 games, giving you the bonus and nothing else. You also had less stats on higher level, which essentially proves you're doing less than compared to lower rating (which makes sense)

Might be an old record, but I don't think 50 games during the bonus every season is gonna help a lot in trying to determine where you are. You're measuring an account with low games vs a settled account afterall.

Just a theory though.

edit @above - harsh yo.


I chose 50 games because that is when the SR bonus wears off (before season 6) so that is where SR can be reasonably be assumed to be close to MMR. You'll note above that I also post numbers for first 50 games in the silver account, for comparison.

But yes, there are a lot of little (and big) details that prevent this from being a pure test. At some point though, I just want to play the game and I'd rather my main account was higher (so I push it harder) because that is where most of my prestige and cosmetics are.
08/25/2017 07:33 AMPosted by Aerohank
I don't suppose you ran this experiment blind?


Heh, No. I don't have the resources to do a proper scientific study. For that, Blizzard would have to hire me and give me a budget to hire subjects.

I kinda would like a feature though that hides all evidence of SR when playing competitive, and just reveals it at the end of the season. I get excessively stressed about my SR, especially when I'm at a promotion boundary, which probably contributed to me bouncing off gold on my silver account and plat on my gold account.

Of course, for it to be really blind I'd also need a way to prevent me form knowing what account I'm on, which is a feature that Blizzard has no reason to implement.
08/25/2017 10:42 AMPosted by Kaawumba
Heh, No. I don't have the resources to do a proper scientific study. For that, Blizzard would have to hire me and give me a budget to hire subjects.

I once read a post that it only takes a handful of games for a player to get to their true SR, and at the very least your experiment suggests that it would take a whole lot more. This is good to know, and I thank you very much for that.
I think the only valid reason out of your list is "skill required to succeed at different tiers is vastly different". It makes sense, a very good diamond shotcalling healer main is not going to win games in bronze where its just random death matches and no one goes for objective......they would need to swap to dps and kill things in order to get out of bronze...


QFT.

I had this issue last season, maining Mercy, when I ranged from 1300 to 1700 SR. Sometimes I'd get as high as 1900, but I'd always drop to 1700. This was despite my 60% win rate (Post Orisa patch SR gains ftw...).

Then I mained Torb. Played rubbish at first, but still went up as I gained more SR by being carried as Torb in a zero medals win than I did with an awesome Mercy win (Huzzah for hero specific SR...).

Got up to 1900, noticed my Torb was not good enough, swapped back to Mercy and got to 2008. Team standard was a lot better and I could support better. More team fights occurred and I could heal more, ult faster and do more for the team.

This season I started to main Mercy again, but was frustrated by how inept some of my team mates were (e.g. Idiot DVa 1v4 entire enemy team blames me for getting de-meched. Get called "!@#$ing noob!" etc. Most upsetting). So went back to Torb. Am now close to 2300 SR.

Sometimes if you want a job done, you just gotta do it yourself...
08/25/2017 10:34 AMPosted by Kaawumba
I'm well aware that the only consistent factor in my games is me, and if I was better my rank would be higher (in both accounts). That isn't really what the post is about. The post is about the reliability (or lack thereof) of SR as a measure of player ability. I consider the game to have the similar mechanics to poker. Better players will win more over the long haul, but in the short-term your cards (other players) can turn against you or save you from your mistakes.


That's a short term mindset.

Short Term - The numerical value associated to your name.

Long Term - Self improvement.

You shouldn't care about the number next to your name but how well you can read your opponents. The SR holds no value/meaning because people like yourself have no impact/no value in the matches that you play. Whether you play Bronze or whether you play Masters, you gain nothing out of them. Especially if you think if it's similar to a poker match where it depends on the cards you are dealt.

99.5% of the people in competitive want to win, but they want to win for all the wrong reasons.
- Hit "next rank here"
- 1 Win away from "next rank"
- That dudes a troll, don't let him win
- I've lost so many games, I need to win this one

Rarely do you get correct assessments.
- Nice Deadeye McCree (Lucio booped 4 into it).
- Good Heals ( Tanks are preventing 95% of their damage intake).

When I look through the gold to Diamond matches I've played, there is a player skill difference and you can tell who in my or their team is not at the correct level. If you can't see some form of difference whether it be mechanical aim or game sense between your 2 different ranks then you aren't making an impact in your games. You don't even know what you're doing right or doing wrong.
I'm well aware that the only consistent factor in my games is me, and if I was better my rank would be higher (in both accounts). That isn't really what the post is about. The post is about the reliability (or lack thereof) of SR as a measure of player ability. I consider the game to have the similar mechanics to poker. Better players will win more over the long haul, but in the short-term your cards (other players) can turn against you or save you from your mistakes.


I've played against players of 1700SR to 2800SR this season, and I've definitely noticed a difference...

vs 1700 (When I duo queued with my bronze friend earlier in season). Utter stomping. 3 golds each game. Even went 31-1 in one game. Turret is an unstoppable death machine.

vs. 2800 (When I quickplay and my QP MMR is apparently that high so I'm facing and teamed with mid-high plats). I get wrecked. Tracer just murders me every time we meet. Turret never gets built as I am pressured every second. Still win a few as I can (barely) cope vs. other characters, and I have good team mates as well, but get bronzes at best, and sometimes no medals at all. Is far more intense and difficult, for sure.
08/25/2017 03:04 PMPosted by Takoyakisan
You shouldn't care about the number next to your name but how well you can read your opponents.
Sure, but we're talking about two different issues here. What you're describing is how to get better under the system as it currently works. What OP is saying is that the system currently isn't very good at it's job (i.e. producing fair matches), in no small part because in matches 1000 SR apart the quality of the teams he's playing with/against is so inconsistent that his negligible effect on the outcome isn't detected by the system.
I have better stats on this account at 2800-2900 than I do on my main at 2400.

Better team mates will help you do more.
Not really directed at OP in particular, but it kind of begs the question, how much of an SR variance is acceptable? 500sr? 300sr? 0?

Keep in mind that the correlation between SR and skill isn't linear, meaning a 500sr difference isn't worth the same depending on the tier.

I could never imagine a 4k player being stuck at 3.5k, the difference in skill between the 4k player and 3.5k players he/she would be playing against is absolutely massive.

However, i could see that a 2.2k player can get stuck at 1.7k. The difference in skill isn't quite as big.

So before people run these alt account experiments, what kind of variance do people expect? and how much of a variance are they willing to accept?

Join the Conversation

Return to Forum