What is the maximum speed of downloading data via api?

API Discussion
My internet speed is 10Mbit/s (1.2MB/s). I just want know does it make sense to switch to a tariff with a higher speed
I'd guess it's probably akamai cached or it uses their own servers in their datacenters or something similar, so think multiple Gb/s connections. I have no clue, at least nothing any of us can probably afford privately.

The main question is; why upgrade your internet for API calls...?
If you do that many calls and performance is an issue you will probably want to rent a server (private, shrared, vm, etc. depending on your needs/budget).

Besides good chance that if performance is that much of an issue, your 10Mb line will not be the issue. There are a lot of ways to optimise the scenario of API data collection from a software point of view.
performence not an isue, data parsed fast (less than 1sec). i download wow auction dumps (5-30mb files), and it takes 3-60sec, so download speed is bootleneck for me
P.S. i dont think thay use cdn, downloading us realm (i tried medivh) data is mach longer, so one data center for region
03/20/2017 03:48 AMPosted by Анусай
performence not an isue, data parsed fast (less than 1sec).

I meant performance in general; with the file sizes taking a long time to download, it degrades application performance.

And some examples of what I meant with software/design solutions for dealing with performance:
- streaming HTTP, parsing elements/objects the moment they come in without waiting for the whole AH document
- Using the last-modified field in the API request for the file to see if you need to redownload it
- etc.

But in case you want frequent updates of every AH, I'd suggest to just run it on a hosted server. Else, if you don't want to go hosted or switch to things like streaming parsing, in terms of internet speed get anything you can afford/want.

If you want numbers: I just ran completely unreliable wget tests on 2 of my servers (at different providers, 1 hosted @ OVH northern france, other inside a vpc @ Azure West-EU):
server@ovh:
auctions.json 100%[===================>] 24.90M 3.69MB/s in 8.4s
2017-03-20 12:46:33 (2.96 MB/s) - ‘auctions.json’ saved [26111531/26111531]

server@azure:
auctions.json 100%[===================>] 24.90M 2.73MB/s in 11s
2017-03-20 12:47:15 (2.18 MB/s) - ‘auctions.json’ saved [26111531/26111531]


These figures actually don't mean anything (due to routing, congestion, active load, etc.), but can give you an impression.

(p.s. you're right, it's not a public webcache, but their own datacenter, atleast in EU, IP is registered to blizzard)
tnx for help
03/20/2017 03:48 AMPosted by Анусай
i download wow auction dumps (5-30mb files), and it takes 3-60sec, so download speed is bootleneck for me

Download using gzip transfer-encoding. It reduces the download size by 90%.

EU chamber-of-aspects data file 9668553 bytes (transfer length 891089, 9.2%), 1.22sec, 729KBps

EU die-aldor data file 8438825 bytes (transfer length 767016, 9.1%), 1.06sec, 723KBps

EU deathwing data file 9235275 bytes (transfer length 775266, 8.4%), 1.82sec, 427KBps

EU dragonmaw data file 5307247 bytes (transfer length 463582, 8.7%), 1.25sec, 371KBps

EU burning-blade data file 10418727 bytes (transfer length 864397, 8.3%), 1.2sec, 720KBps

EU frostmane data file 7812858 bytes (transfer length 685561, 8.8%), 0.98sec, 702KBps

EU darkmoon-faire data file 4235755 bytes (transfer length 368272, 8.7%), 1.05sec, 349KBps

Speed comparison:
$ curl 'http://auction-api-eu.worldofwarcraft.com/auction-data/4a7ab717fe035ffc4febb7650f84de3a/auctions.json' | wc -c

9018k - 0:00:15
9235275 bytes

$ curl -H 'Accept-Encoding: gzip' 'http://auction-api-eu.worldofwarcraft.com/auction-data/4a7ab717fe035ffc4febb7650f84de3a/auctions.json' | gunzip | wc -c
757k - 0:00:02
9235275 bytes
03/20/2017 08:17 AMPosted by erorus
Download using gzip transfer-encoding. It reduces the download size by 90%.
wow tnx, u save my money)

Join the Conversation

Return to Forum