Diablo® III

AT&T is killing your latency, graphs inside

http://imgur.com/a/GiwIN

I obscured Comcast hostnames that more specifically pinpoint my location in FL, other than that these are not edited in any way. Since I just started recording data a few hours ago, this is not a complete picture, but it is a picture of what a few hours of sampling every minute with traceroutes is demonstrating about AT&T's network leading in to Blizzard's.

Methodology:

* Using xmlroute to run traceroutes to 12.130.244.193. This is not the "official" IP I saw in some other threads, but it is the IP that the Diablo III process connects to consistently when viewing Windows' TCP connections.
* Using background Windows task to run xmlroute and pipe output to a file on disk every minute.
* Using Splunk to pickup the data in that file, with each hop delimited as an event.
* Using Splunk queries to extract the data and visualize it as seen in the imgur url.

In general, today is a reasonably good latency day for me, though it IS a bit spikey. I can fire nether tentacles right now and won't see it render for a second or two every few minutes. The important take away there is that where you see high latency in my graphs, you should know that the actual latency is quite a bit higher thanks to averaging.

In the horizontal bar chart, the bottom 4 hosts are my router and Comcast's routers. You can see that over the last few hours, not only is ATT spikey, but it's also 2-3x worse than Comcast.

Problematic hosts in AT&T's network:
gar29.la2ca.ip.att.net
12-122-254-238.attens.net
12-122-254-234.attens.net
Occassionally cr2.la2ca.ip.att.net

When you look at my timechart you can see some hosts do spike on occasion, but gar29.la2ca.ip.att.net is a consistently bad performer over time.

Just wanted to bring in some data over time into the latency discussion as the spikey nature of ATT's latency sometimes prevents point-in-time traceroutes from revealing a complete picture.

For the blues, should we as end-users be reaching out to ATT on this issue? I assume Blizzard knows where packet loss or congestion is happening and is working with ATT on it.
Edited by Thatonyg#1499 on 6/17/2012 4:45 PM PDT
Reply Quote
I've been having the same issue. I am also with Comcast Cable. My tracerts routinely show high ping times inside the AT&T network. As I am not an AT&T customer, I feel like it is Blizzard's responsibility to get the issue resolved, not ours. If I recall correctly, AT&T is their service provider anyways. But yeah, I haven't been able to play the last few weekends at all because of this. I'm sick of it.
Reply Quote
Yeah, after the attens.net IP addresses traces go silent/dead, so it's hard to tell for sure... but Blizzard is probably colocated with AT&T Enhanced Network Services (attens) and at least some of those IPs likely belong to Blizzard hosts.
Reply Quote
Also, I agree with you, since I'm not a customer of AT&T I don't think I'd get far in reporting an issue, and would prefer that Blizzard aggregate the voice of the customer and use their higher level contacts to get something done about gar29 and other problematic hops.
Reply Quote
12.129.199.182 = CERF.NET, ATT ENS.

that is what is after.

i created a post about this: http://us.battle.net/d3/en/forum/topic/5149182570

however, recently, the lag on the att routers has dropped off, or at least during periods of high 'apparent' latency when i checked recently. i say this because ping is hanging around 200, but it feels more like 2000. ;o it seems to be all on blizzard's network now, congestion on their servers most likely.

and ugh, imgur is down again :\
Edited by betatrash#1974 on 6/17/2012 5:41 PM PDT
Reply Quote
- Technical Support
View profile
Very very good post. Constructive and backed by data. *cheers* They could at least acknowledge the issue, if you are correct, and say they are working with the provider to resolve it.
Reply Quote
Amen, alleluia, desperately awaiting a response. Free bump here.
Reply Quote
12.129.199.182 = CERF.NET, ATT ENS.

that is what is after.

i created a post about this: http://us.battle.net/d3/en/forum/topic/5149182570


Good thread and solid info -- I bumped it for you. Not many people know the difference between what happens with ICMP 8 and ICMP 11 :)


however, recently, the lag on the att routers has dropped off, or at least during periods of high 'apparent' latency when i checked recently. i say this because ping is hanging around 200, but it feels more like 2000. ;o it seems to be all on blizzard's network now, congestion on their servers most likely.


Yeah, the game feels a little bit better, but I agree, when it says 200ms it doesn't feel like any other game I've played online at 200ms. It feels much slower, probably because the problematic hosts basically have "microstutter" at a network level. One packet goes through at 500+ms and the next 3 at 100ms and the blended average looks OK, but the game is not playable.

Here's an updated graph of gar29's ongoing performance problems. As you can see some of the other routers calmed down but gar29 is still as unpredictable as ever. Interestingly one of the other problematic hosts (12-122-254-234.attens.net) is occassionally not registering in Splunk at all (second chart in the url below) -- it may simply be losing packets or timing out.

http://imgur.com/a/hfS2O


and ugh, imgur is down again :\


Looks like it's back up, let me know what you think as I'm very interested in your thoughts.
Edited by Thatonyg#1499 on 6/17/2012 7:07 PM PDT
Reply Quote
some of the routers are load-balanced. half the packets will return from 1 ip, half from the other, maybe thats causing an issue? not sure, i made my own combination tool of ping/tracert, called it TTLping just to diagnose this problem. the info is buried in the thread.

as for imgur, must be a routing issue. down over here.

i have an open case with ATT ENS, never bothered to call in. don't know how forthcoming they will be though...
Edited by betatrash#1974 on 6/17/2012 7:32 PM PDT
Reply Quote
bypassing my local dns here, by IP, cant connect to imgur. lolz. probably my isp's dns is cached to the wrong ip, oh well, cant be bothered atm ;p

You've requested a page on a website (141.101.125.192) that is on the CloudFlare network. Unfortunately, CloudFlare is currently unable to resolve your requested domain (141.101.125.192). There are two potential causes of this:
Edited by betatrash#1974 on 6/17/2012 7:38 PM PDT
Reply Quote
Re: load balancing, could be the problem, though that is the only router in the trace that appears intermittently. I should be getting at least 1 additional intermittent host if they are alternating routes.

Also, I got the same error when I went to imgur by IP, but it works fine by hostname. CloudFlare must be looking for a valid Host: header in the http request?
Reply Quote
bypassing my local dns here, by IP, cant connect to imgur. lolz. probably my isp's dns is cached to the wrong ip, oh well, cant be bothered atm ;p

You've requested a page on a website (141.101.125.192) that is on the CloudFlare network. Unfortunately, CloudFlare is currently unable to resolve your requested domain (141.101.125.192). There are two potential causes of this:


If you're connecting by IP, it's probably not a DNS issue ... (Edit: I feel like you've gone on a tangent and misworded something and actually know what you're talking about)

To the OP: You may have half my Internets
Edited by titegtnodI#1873 on 6/17/2012 7:52 PM PDT
Reply Quote
85 Draenei Paladin
10140
Thanks for putting together the info. So basically, our route to the D3 servers is basically screwy, right?

I guess I'm confused about that, because for a lot of us, things were perfect before X patch was launched. IE., my lag was perfect until 1.02 on May 29th, and only D3 is giving problems. As soon as 1.02 came down, all kinds of crazy rubberbanding, etc. going on.
Edited by mendhi#1900 on 6/17/2012 8:04 PM PDT
Reply Quote
Thank you guys for the support.

Mendi, I can't speak for everyone for certain, I don't know ATTENS' topology or Blizzard's. Chances are everyone routes through those bad ATT hops though.

The patch could have created problems if they changed their netcode on the client or the server, I'm not really sure. The one thing that *is* certain is that at a basic networking level, there are hops inside the ATT network (some of which may belong to Blizzard) that appear to either be dropping packets or appear to be congested/overwhelmed.
Reply Quote
I really tried to understand all the technical stuff, but I only could grasp some of it. I'm still learning, yay me. Anyway, on to the topic at hand... I use AT&T for everything in my home, it's all bundled through AT&T U-verse. I wasn't having any latency issues with Diablo 3 until recently. In the past I might get a small hiccup, but not the amount I am receiving now. Now when I try to play I'm consistently reaching 1k+ latency. Is there any action I can take currently to alleviate this? Should I run a trace? And if yes, how do I start?
Reply Quote
I really tried to understand all the technical stuff, but I only could grasp some of it. I'm still learning, yay me. Anyway, on to the topic at hand... I use AT&T for everything in my home, it's all bundled through AT&T U-verse. I wasn't having any latency issues with Diablo 3 until recently. In the past I might get a small hiccup, but not the amount I am receiving now. Now when I try to play I'm consistently reaching 1k+ latency. Is there any action I can take currently to alleviate this? Should I run a trace? And if yes, how do I start?


You can run a traceroute by doing the following:
Open a command prompt in Windows (doesn't have to be as admin)
Type tracert 12.130.244.193

You should see each hop, along with 3 latency numbers which are point-in-time snapshots of the round trip time for a very small packet, up until you reach Blizzard's network at which point the traceroute will tell you it's getting * for the latency number (no response).

The tricky thing is that the measurements are just point-in-time snapshots, so if you have a device dropping or deprioritizing packets occassionally, you might not see it with a single traceroute. Try running a few, and definitely try during peak and when you're having an ongoing latency issue.

Note, border routers (the hand-off between Comcast and AT&T for example) are notoriously slow(er) -- a lot of networks will promise something like 40ms anywhere in their network, but once you traverse from one network to another your performance is a gamble. People on AT&T services may have fewer issues if they aren't crossing any borders.
Reply Quote
I did a bit more research on the problematic hosts (gar29.la2ca.ip.att.net).

There are several posts on various forums from WoW players who also route through those hosts.

Good news: if those hosts are dropping packets or causing burst lag for us, they must be doing the same for WoW, which means Blizzard is most definitely aware of it and our chances of something being done increase.

Bad news: these are probably behemoth routers that can't simply be replaced with something else they have sitting on the shelf.

I also have updated graphs that show activity overnight as well if anyone is interested.
Reply Quote
well, one problem with icmp is that QoS will bump it to the bottom of the priority list. so those aren't exactly the response times for other traffic.

however, when a router starts delaying icmp responses it's usually because it is busy doing something else. but it isn't really a 'true' representation of the ping.

i see way more problems on the blizzard servers now that i do on those att routers.
Reply Quote
Yeah, a pure tcp measurement would be better. I'd love to be able to believe the in-game latency statistic but it's misleading at best.

Agree that the servers are part of the problem but if they are shaping/deprioritizing udp with these kinds of fluctuations they must be really, really busy and trying to run the service very close to capacity.

I'd be OK with a consistent 300ms too, but it's just all over the place. It's like trying to play a shooter when framerates are fluctuating wildly, it throws your sense of timing off... and so much about this game is timing.
Reply Quote
Okay so I did the trace, and this is what I found...

Tracing route to 12.130.244.193 over a maximum of 30 hops

1 2 ms 2 ms 2 ms home
2 34 ms 57 ms 22 ms 76-219-252-3.lightspeed.sndgca.sbcglobal.net [76
.219.252.3]
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 23 ms 24 ms * 12.83.70.141
7 27 ms 27 ms 28 ms gar29.la2ca.ip.att.net [12.122.129.241]
8 28 ms 27 ms 28 ms 12-122-254-234.attens.net [12.122.254.234]
9 27 ms 28 ms 28 ms 12.129.193.254
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.


That seems like an awful lot of time outs.... The latency on the servers right now looks ok, I will try it again if I start having issues.

EDIT: Loaded up Diablo 3 after the post, and it is totally unplayable. I can't even leave the game, the latency is so horrific. Ran another trace route, with similar results. The highest latency was for gar29.la2ca.ip.att.net --> 36 ms. So are those numbers bad? Or is there something else going on? I'm so disappointed. I finally have a full day off and I can't even enjoy it by playing Diablo....
Edited by Vrishka#1843 on 6/18/2012 1:57 PM PDT
Reply Quote

Please report any Code of Conduct violations, including:

Threats of violence. We take these seriously and will alert the proper authorities.

Posts containing personal information about other players. This includes physical addresses, e-mail addresses, phone numbers, and inappropriate photos and/or videos.

Harassing or discriminatory language. This will not be tolerated.

Forums Code of Conduct

Report Post # written by

Reason
Explain (256 characters max)
Submit Cancel

Reported!

[Close]