Problem with AT&T Backbone and NY Data Center

Technical Support
So I've been having these huge problems the last week with my connection. For some reason it had gone to hell, average ping of 100 went up to 300. Instantaneous ping with wireshark shows pings up to 3000-4000 (packetloss/spiking). I went to TWC first to track down the problem and they were very helpful trying to reprovision the modem and such to a different path. We determined that the flaw lies on the AT&T Backbone that is taking the network traffic off my ISP's network and to Blizzard's network.

Here are the results of a tracert from me to Arygos:

Tracing route to 206.17.111.81 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.1
2 9 ms 11 ms 9 ms cpe-76-169-40-1.socal.res.rr.com [76.169.40.1]
3 7 ms 10 ms 8 ms cpe-76-166-12-57.socal.rr.com [76.166.12.57]
4 14 ms 12 ms 12 ms 66.75.159.80
5 13 ms 15 ms 15 ms BE20-lsanca4-rtr3.socal.rr.com [66.75.159.128]
6 13 ms 15 ms 12 ms BE25-lsanca4-rt1.socal.rr.com [66.75.161.156]
7 13 ms 14 ms 16 ms ae-6-0.cr0.lax00.tbone.rr.com [66.109.6.212]
8 16 ms 20 ms 14 ms ae-0-0.pr0.lax10.tbone.rr.com [66.109.6.133]
9 23 ms 17 ms 17 ms Vlan511.icore1.EQL-LosAngeles.as6453.net [206.82
.129.9]
10 18 ms 14 ms 15 ms if-4-28.tcore2.LVW-LosAngeles.as6453.net [216.6.
84.53]
11 24 ms 33 ms 23 ms if-2-2.tcore1.LVW-LosAngeles.as6453.net [66.110.
59.1]
12 14 ms 16 ms 17 ms 192.205.35.129
13 174 ms 172 ms 172 ms cr2.la2ca.ip.att.net [12.122.129.34]
14 168 ms 168 ms 162 ms cr1.slkut.ip.att.net [12.122.30.29]
15 165 ms 168 ms 168 ms cr2.dvmco.ip.att.net [12.122.30.26]
16 89 ms 88 ms 90 ms cr1.cgcil.ip.att.net [12.122.31.86]
17 165 ms 163 ms 165 ms cr2.cgcil.ip.att.net [12.122.2.54]
18 90 ms 89 ms 89 ms cr1.n54ny.ip.att.net [12.122.1.1]
19 167 ms 168 ms 160 ms cr2.n54ny.ip.att.net [12.122.2.14]
20 90 ms 89 ms 90 ms cr1.cb1ma.ip.att.net [12.122.31.126]
21 167 ms 166 ms 168 ms gar5.cb1ma.ip.att.net [12.122.145.33]
22 171 ms 171 ms 168 ms 12-122-254-14.attens.net [12.122.254.14]
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

The Problem clearly arises at this point:
13 174 ms 172 ms 172 ms cr2.la2ca.ip.att.net [12.122.129.34]
until
22 171 ms 171 ms 168 ms 12-122-254-14.attens.net [12.122.254.14]


At some point at or between these points there is a severe bottleneck causing a ping jump. If I spam tracerts for a 24 hour period these pings fluctuate anywhere from 170 to 3000 (sometimes showing no response and packetloss). I can't tell if its a specific node or the entire backbone being overloaded, but there is clearly an issue on the Level 1 transportation route of Blizzard traffic from my TWC connection to their NYC Datacenter.

For kicks I also decided to use smoothping to see if it would take care of my problem by redirecting the traffic through a different route, and it did. A smoothping Tracert to their servers results in:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 9 ms 9 ms 9 ms cpe-76-169-40-1.socal.res.rr.com [76.169.40.1]
3 7 ms 9 ms 12 ms cpe-76-166-12-57.socal.rr.com [76.166.12.57]
4 12 ms 12 ms 12 ms 66.75.159.80
5 13 ms 12 ms 13 ms BE20-lsanca4-rtr3.socal.rr.com [66.75.159.128]
6 14 ms 12 ms 12 ms BE26-lsanca4-rt1.socal.rr.com [66.75.161.158]
7 15 ms 19 ms 12 ms ae-5-0.cr0.lax00.tbone.rr.com [66.109.6.102]
8 13 ms 13 ms 12 ms ae-0-0.pr0.lax00.tbone.rr.com [66.109.6.135]
9 12 ms 12 ms 12 ms TenGigabitEthernet1-2.ar5.LAX1.gblx.net [64.208.
205.153]
10 119 ms 121 ms 118 ms ACCESSIT-EZZI.net.Vlan1165.asr1.EWR2.gblx.net [6
4.208.169.58]
11 85 ms 86 ms 86 ms 17-110-9-72.reverse.ezzi.net [72.9.110.17]
12 86 ms 86 ms 86 ms ads-bsh-fwc2.ezzi.net [72.9.110.163]
13 96 ms 85 ms 87 ms 72-9-104-146.reverse.ezzi.net [72.9.104.146]

Now this is the same concept, a Los Angeles connection going to a NYC server. Note it does not take the AT&T backbone path but rather the EZZI path. This path has no problems, my ping is 80 and remains that consistently if I tracert for a long period of time.

If I connect to this proxy first and then to the wow server (bypassing the AT&T backbone altogether) I experience a constant world ping of 80 (like it used to be a long time ago) with no instantaneous ping spikes or packetloss issues. This is because my connection is going across EZZI to NYC instead of AT&T. Once it is in NYC it is taking some path from EZZI to the NY Data Center that is not experiencing these issues. This is why the problem does not occur unless you are far enough away from NYC.

Think of it as taking a detour. The AT&T highway is so congested that its better for me to take the EZZI highway instead (even if it might take me out of my way first) and then take another route from that highway's end to Blizzard.

There is a very clear problem here, not on the user end, not on the ISP end but rather on the path that Blizzard's system is configured to send data from the West Coast (and possibly more) to the East Coast where there is not enough bandwidth available. Bypassing this by using a Proxy is an option, but do I really have to pay extra money just to be able to play WoW? The game is completely unplayable during peak hours for me now and my only options seem to be:

1. Blizzard and AT&T fix this
2. I pay for a proxy service
3. I move my characters to a server at a different datacenter

All of which are beyond my control or require me to pay out of pocket for something that is not my problem.....
This sounds like the same problem I am having...Good to see its not just me

http://us.battle.net/wow/en/forum/topic/2228227739
I want to bring as much light to this issue as possible because it has happened on and off before and I am getting sick of it. I have purchased Smoothping for a month just so my wife and I can raid to avoid a problem that is Blizzard's issue. In a month if it is not resolved, we will be either server transferring off the NYC Data Center or contacting our credit card and just dropping the game altogether since they are shirking this responsibility and are having users run through hoops.

Admittedly "usually" it is an issue in the user's end but any novice network technician can clearly see there is an issue on the AT&T backbone.

For anyone not familiar with how the internet works, that portion of the connection is purchased by Blizzard from AT&T to carry the data from your ISP to their network. Think of it like their this, its like an interstate highway and your local ISP networks are all the entrance ramps. As your data leaves your ISP's network it goes onto the highway and to Blizzard's servers. It is their contract and their control, and if there isn't enough bandwidth then it is Blizzard who has to purchase more from AT&T or switch their contract to another backbone provider.

The "ONLY" way this would be out of their control was if their contract is not with AT&T and the company that they are using is effectively "subcontracting" AT&T to transport those portions of the country. I highly doubt this as I'm pretty sure its fact that Blizzard uses AT&T as their Level 1 servicer. This is also reaffirmed by the fact that it goes straight from AT&T into Blizzard and not back onto another Level 1 provider first.

If you are on the West Coast and are connecting with any number of ISPs from Roger, Time Warner, Cox, etc. and you are playing on a server at the NYC data center, you are almost certainly suffering from this. It is not your fault. It is not your ISP's fault. It is the backbone that Blizzard has chosen to use to gather data from those portions of the country.
I consistently drop packets or spike high during the night time on my last two hops to US-Icecrown. Here is just snip of the tracert I ran last night. Unfortunately I don't have the tracert from yesterday morning when I was doing nothing online and these same last two hops were over 600ms and all prior hops were sub 60ms.


12 95 ms 98 ms 104 ms 12-122-254-114.attens.net [12.122.254.114]

13 * 97 ms 96 ms mdf001c7613r0003-gig-10-1.nyc3.attens.net [63.240.65.10]


Edit- Frontier is the ISP. Wired connection. No connections issues before March 6th. I've also read there was a connection issue between Frontier and AT&T someone mentioned here http://us.battle.net/wow/en/forum/topic/2140267777?page=6. Any way we could get a update on that issue ?
For anyone not familiar with how the internet works, that portion of the connection is purchased by Blizzard from AT&T to carry the data from your ISP to their network. Think of it like their this, its like an interstate highway and your local ISP networks are all the entrance ramps. As your data leaves your ISP's network it goes onto the highway and to Blizzard's servers. It is their contract and their control, and if there isn't enough bandwidth then it is Blizzard who has to purchase more from AT&T or switch their contract to another backbone provider.

This actually is not very accurate. I don't really know where you got this idea, but it is unfortunately incorrect.

AT&T (attens) is our provider, but the routing TO us is not controlled by them, at all. The only thing they control is the initial return route. Your trace shows 3 different backbones prior to even reaching our ISP (TWC > TWC backbone > TATA > AT&T > attens), and the problem appears at the peering hop from TATA to AT&T. There's no way to tell from a customer-initiated traceroute which side of the peering hop the issue is on.

While we have already notified our NOC about this, please be careful when posting unsubstantiated information. All it ends up doing is confusing other people and, potentially, discrediting your post.
________________________________________________
Account and Technical Services || Tues - Sat 645 - 1545 PST
Not seeing a resolution on the forums? Contact a Support Rep directly! - http://us.blizzard.com/en-us/company/about/contact.html

Tigers love pepper... they hate cinnamon.
Let me know how I'm doing! :) - http://www.surveymk.com/s/RMW9Y6X
For anyone not familiar with how the internet works, that portion of the connection is purchased by Blizzard from AT&T to carry the data from your ISP to their network. Think of it like their this, its like an interstate highway and your local ISP networks are all the entrance ramps. As your data leaves your ISP's network it goes onto the highway and to Blizzard's servers. It is their contract and their control, and if there isn't enough bandwidth then it is Blizzard who has to purchase more from AT&T or switch their contract to another backbone provider.

This actually is not very accurate. I don't really know where you got this idea, but it is unfortunately incorrect.

AT&T (attens) is our provider, but the routing TO us is not controlled by them, at all. The only thing they control is the initial return route. Your trace shows 3 different backbones prior to even reaching our ISP (TWC > TWC backbone > TATA > AT&T > attens), and the problem appears at the peering hop from TATA to AT&T. There's no way to tell from a customer-initiated traceroute which side of the peering hop the issue is on.

While we have already notified our NOC about this, please be careful when posting unsubstantiated information. All it ends up doing is confusing other people and, potentially, discrediting your post.
________________________________________________
Account and Technical Services || Tues - Sat 645 - 1545 PST
Not seeing a resolution on the forums? Contact a Support Rep directly! - http://us.blizzard.com/en-us/company/about/contact.html

Tigers love pepper... they hate cinnamon.
Let me know how I'm doing! :) - http://www.surveymk.com/s/RMW9Y6X


I understand that you guys have no control over the routing before attens, I never at any point said that you did. I said that you guys have control over your contract with attens Level 1 services. My points all relate to traffic on attens, not traffic before attens. I guess I should have made it clear that there can be middle-men too that you do not control, but I was keeping the explanation simple and the middle-men are irrelevant at the moment.

I'm glad to hear that communication has been established but this sounds like its going to turn into finger pointing between AT&T and Blizzard again with neither assuming responsibility. As I have mentioned though, the problem is not in the routing TO attens, but rather the routing ON attens. ATTENS IS THE PROBLEM. That is why the issues are experienced across a broad range of ISPs and not any particular one.

I am going to point my finger at the more broad problem though because if you look at the tracerts of other people who have been showing these issues, they are experiencing the issues at different attens backbone locations. It seems like that entire route is just completely bogged down with traffic.

So what happens in the end to us if Blizzard keeps saying it is an AT&T problem and AT&T keeps saying it is a Blizzard problem?

By the way, TWC seems to agree with me the last time we communicated, they said that you are not purchasing enough bandwidth from your Level 1 provider and you need a larger contract to accommodate cross country traffic from West Coast locations.

I'm not trying to be nasty or mis-informative about all of this. I'm just really tired of seeing and being asked by techs to post irrelevant information and run through hoops that clearly do not apply here. I don't see how a ticket for this issue wasn't opened from Blizzard to AT&T by now with at least some official response regarding what is causing it... At the very least you should be able to tell us if it is a temporary issue from AT&T having hardware difficulties or a congestion issue.

If it is a congestion issue (as it appears) then somebody needs to man up and pay to fix it or start offering some free transfers for us to datacenters where this problem does not exist.
03/18/2011 2:32 PMPosted by Darkmgl
I am going to point my finger at the more broad problem though because if you look at the tracerts of other people who have been showing these issues, they are experiencing the issues at different attens backbone locations. It seems like that entire route is just completely bogged down with traffic.

That, too, has been reported to our NOC. They are investigating it with our provider now. :)

03/18/2011 2:32 PMPosted by Darkmgl
By the way, TWC seems to agree with me the last time we communicated, they said that you are not purchasing enough bandwidth from your Level 1 provider and you need a larger contract to accommodate cross country traffic from West Coast locations.

I heard that rumor. It is unequivocally inaccurate, misleading, and incorrect. I debunked that theory in another thread. We are nowhere near capacity, even at peak... not even close. In fact, we could halve our leased pipes and still be 50% under capacity.

What exactly does this mean? We need more bandwidth to accommodate traffic that travels farther? How does that work? I would love for those you are in contact with to contact our NOC directly; they would love to open a dialogue and... discuss these things.

Regardless, your ISP's direct Tier 1 provider has, in fact, already been speaking with our NOC directly regarding another issue in recent months, and I know for a fact they are not under the impression that we are oversaturated. Any way, it is very disappointing that a third party would be making uninformed accusations like that, if that is indeed the case. I would expect a bit more professionalism in cases like this, as they have no way of knowing exactly what our current capacity is. To even speculate otherwise is irresponsible.

I'm not trying to be nasty or mis-informative about all of this. I'm just really tired of seeing and being asked by techs to post irrelevant information and run through hoops that clearly do not apply here. I don't see how a ticket for this issue wasn't opened from Blizzard to AT&T by now with at least some official response regarding what is causing it... At the very least you should be able to tell us if it is a temporary issue from AT&T having hardware difficulties or a congestion issue.

I share your fatigue with situations like this, but the nature of the internet, especially in this new era of streaming HD content, leads me to believe that this will be an ongoing fight until certain conditions change. Everyone (especially tier 1s) is feeling the strain. Unfortunately, exact details of discussions occurring behind closed doors, including the content of ongoing troubleshooting between third parties, will probably never be posted in a forum such as this.

At least not by me. :)

03/18/2011 2:32 PMPosted by Darkmgl
If it is a congestion issue (as it appears) then somebody needs to man up and pay to fix it or start offering some free transfers for us to datacenters where this problem does not exist.

I wish you the best of luck in getting a Tier 1 to pay for your character transfer, but Blizzard's policy has always been that if the issue is beyond our control, we cannot offer any form of compensation. As always, you are free to contact a Billing and Support representative for assistance, but I would not encourage such a course of action.

o/
________________________________________________
Account and Technical Services || Tues - Sat 645 - 1545 PST
Not seeing a resolution on the forums? Contact a Support Rep directly! - http://us.blizzard.com/en-us/company/about/contact.html

Tigers love pepper... they hate cinnamon.
Let me know how I'm doing! :) - http://www.surveymk.com/s/RMW9Y6X
I am going to point my finger at the more broad problem though because if you look at the tracerts of other people who have been showing these issues, they are experiencing the issues at different attens backbone locations. It seems like that entire route is just completely bogged down with traffic.

That, too, has been reported to our NOC. They are investigating it with our provider now. :)

By the way, TWC seems to agree with me the last time we communicated, they said that you are not purchasing enough bandwidth from your Level 1 provider and you need a larger contract to accommodate cross country traffic from West Coast locations.

I heard that rumor. It is unequivocally inaccurate, misleading, and incorrect. I debunked that theory in another thread. We are nowhere near capacity, even at peak... not even close. In fact, we could halve our leased pipes and still be 50% under capacity.

What exactly does this mean? We need more bandwidth to accommodate traffic that travels farther? How does that work? I would love for those you are in contact with to contact our NOC directly; they would love to open a dialogue and... discuss these things.

Regardless, your ISP's direct Tier 1 provider has, in fact, already been speaking with our NOC directly regarding another issue in recent months, and I know for a fact they are not under the impression that we are oversaturated. Any way, if is very disappointing that a third party would be making uninformed accusations like that, if that is indeed the case. I would expect a bit more professionalism in cases like this, as they have no way of knowing exactly what our current capacity is. To even speculate otherwise is irresponsible.

I'm not trying to be nasty or mis-informative about all of this. I'm just really tired of seeing and being asked by techs to post irrelevant information and run through hoops that clearly do not apply here. I don't see how a ticket for this issue wasn't opened from Blizzard to AT&T by now with at least some official response regarding what is causing it...

I share your fatigue with situations like this, but the nature of the internet, especially in this new era of streaming HD content, leads me to believe that this will be an ongoing fight until certain conditions change. Everyone (especially tier 1s) is feeling the strain. Unfortunately, exact details of discussions occurring behind closed doors, including the content of ongoing troubleshooting between third parties, will probably never be posted in a forum such as this.

At least not by me. :)

o/
________________________________________________
Account and Technical Services || Tues - Sat 645 - 1545 PST
Not seeing a resolution on the forums? Contact a Support Rep directly! - http://us.blizzard.com/en-us/company/about/contact.html

Tigers love pepper... they hate cinnamon.
Let me know how I'm doing! :) - http://www.surveymk.com/s/RMW9Y6X


Thats cool at least. I would expect at this point some of the lack of professionalism would be due to the amount of complaints TWC receives about this. I know on the DSLReports forums the live techs have been getting bombarded with this stuff for months and I'm sure they are sick of hearing about it. Theres been more than enough finger pointing from Blizzard to TWC over the last year or two too from this forum stating that it is a TWC issue. I'm sure TWC is none to happy having customers call and threatening to cancel because of that.

If you guys really are only sitting at 50% capacity during peak hours then that would definitely point at a problem with attens hardware configuration.... Someones gotta pressure them hard to resolve that then.

You are absolutely right though in that the rules and laws for these systems do need to undergo some serious changes. Its such a logistical mess right now trying to figure out who is responsible for what as the data moves across networks.

As for the character transfer, I still do not see where this is not a Blizzard problem. You guys still pick and choose who you make contracts with and the only cost of a character transfer for you guys is the opportunity cost that we would have to pay for it otherwise.... I'm just hoping deep down that if nobody wants to resolve the issue seriously that maybe Blizzard would man up a bit and give their customers a fix that they are capable of, showing some good public relations.
Darkmgl

The IP address that you ran the tracert to is actually an old IP for that server. When you get the chance please run a tracert to this IP instead: 199.107.6.23
________________________________________________
Account and Technical Services Forum Representative
If you can't find a resolution to your issue, feel free to contact a Support Representative directly for further assistance. Their contact information can be found at the following link: http://us.blizzard.com/en-us/company/about/contact.html

I'm available on the forums Monday through Friday from 12PM to 8Pm Pacific Time
Feel free to rate my services at the following link: https://www.surveymk.com/s/RZ289DL
hey, possible reasons, replies and angry blues. good thread.
I'm pretty sure there is something you, or the public relationship sector in your company can help with AT&T issues. So what I'm asking is that, perhaps you can run to knock at the door of their offices and toss a piece of *beep and bring the message, make it clear to them that It's me who wanna say "You are fired!"
So ... are we going to be kept in the loop as to this current AT&T debacle or are are we gonna be treated like mushrooms ... kept in the dark and fed bull**** ?

It's been two months since this issue with AT&T began and this is the FIRST I've read anything on these Forums about the underlying cause. I would have thought that Blizzard would have a pretty good idea of exactly what the issue was after the 1st week. Blizzard walking on eggshells so as not to anger AT&T seems to be the order of the day here.

Instead, I've seen post after post where customers with latency problems are being directed to do this & do that to their computer, contact their ISP, etc ... with NO resolution whatsoever.

Give us the information that points to the problem in a Blue Sticky so we can cut & paste it in an angry letter to noc @ att.net so they will know that WE know who the culprit is.
You are right Vladimir. I've been tracking this particular issue since I experienced after 4.0.6. I believe that besides the super intelligent support on this forum many of us customers are capable to understand various issues as precisely as they do in the past few years, or even capable to help ending this through their channels.

What I know is that there is a two-years' contract with AT&T, and it should not be so hard for blizzard to reclaim. At least, show attitude please.
The reality of the situation is that the people at Blizzard whom are involved in this know so much more about what's going on than you do that you couldn't begin to comprehend any of the technology they're using.

It's in your best interest to not sound like idiots unless you truly know what you are talking about. So far, I haven't ever read any coherent post by any person who could potentially be considered an authority on matters of the internet by anyone who is experiencing trouble.
Excuse my English.

The two weeks I have had the same problem with lag. Unfortunately trace the route to check something on a ping ip 192.205.37.93 very exactly related to AT&T.

IP: 192.205.37.93
server location: United States
ISP:AT&T Services

I am not accusing anybody, but no use in asking for storing firewall, router, etc and it was working normally until then.
The reality of the situation is that the people at Blizzard whom are involved in this know so much more about what's going on than you do that you couldn't begin to comprehend any of the technology they're using.

It's in your best interest to not sound like idiots unless you truly know what you are talking about. So far, I haven't ever read any coherent post by any person who could potentially be considered an authority on matters of the internet by anyone who is experiencing trouble.


Well said.

I have always felt that anyone who comes to the conclusion that they didn't have a problem before the release of a patch = patch must be the problem. Is an idiot.

That isn't to say a patch may not have been the cause of a problem in the past. But you can't just automatically blame a patch just because it came out at the same time theproblem started.

Sometimes they may know the cause of the problem. But the solution is hard to come by. Or any solution that is thought up will cause another problem and thus angering other people.

Well I hope it gets fixed soon because I can't play at all.


when do you think it will be fixed? next reset or patch?

Davidedl?













































Well I hope it gets fixed soon because I can't play at all.


when do you think it will be fixed? next reset or patch?

Davidedl?


You are not going to like this answer but

It will be fixed when it is fixed.

People complaining about how Blizzard doesn't care will not make the problem go away any faster.

My suggestion to those who are having a problem. Either try a different realm in a different data centre. Or put WoW away and go play something else for a while.

I do apologize for being so blunt :)
Mordaro

Your right..... I may not get a answer.

I might just go to lowerping if it doesn't get fixed soon..

Time for a break :D
I've been dealing with this problem for months, it sucks and I hope it's eventually fixed playing on the servers affected by it is most annoying for prime gameplay

Join the Conversation

Return to Forum