Max Concurrent GET Requests

90 Human Paladin
14880
Hey all,

I was working on serializing some JSON from the item API to build an item cache based on a few of the item properties in the JSON that I cherry picked to my liking.

Anyways, off-hand I've no way to ascertain the upper limit on item id's, so I just assume it to be 100k.

So, I fire off 100k asynchronous tasks to perform the GET requests on every item id URL (1 through 100k). Each task then upon completion updates my db with the necessary queries (queries are pooled on simultaneous connections). Assuming 200 or 404 response, if 503 we keep trying, task bails on any 500.

Anyways, I have a few main questions I couldn't find answered in the API docs:

1. I'm not taxed or limited in anyway by the throttling if I hit the item api, as they are static... correct? (In regards to the 3k request/day limit).

2. What's the max concurrent GET requests I can make of the blizz server? Specifically regarding item API.

3. Is there a JSON I can consume of all the items, rather than making 100k async GET requests?
Edited by Laurabelle on 10/1/2013 6:36 AM PDT
Reply Quote
Web & Mobile Team
1. Anything to an /api url should be throttled according to policy. The only thing currently not is the auction house data files (which are served from /auction-data )

2. We don't have a limit. Anything that's at a dangerous level will attract attention and possible action from our server admins.

3. There is not... we want to provide this but don't have a good estimate due to other priorities. The community might have some made you can use.
Reply Quote
90 Tauren Shaman
14635
It occurs to me that #3 is exactly the sort of thing that should be in a git repo somewhere. Anyone have a github account they can volunteer?
Edited by Oryx on 10/16/2012 5:44 PM PDT
Reply Quote
90 Human Paladin
14880
So regarding #1, I assume that each request on the item api is worth less "points" toward the 3k, as I can obviously make far more than 3k item look-ups via the api from a single app.

To put things in perspective, this "item cache" is built once, and then possibly updated periodically, but definitely not an every hour or day sort of thing.

I was using C# as a testbed, and it limits the max concurrent connections to 2 when hitting the same server. This setting can be tweaked in the app.config file, for example:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.net>
<connectionManagement>
<add address = "http://us.battle.net" maxconnection = "2" />
</connectionManagement>
</system.net>
</configuration>


I tested by hitting the first thousand item URLs three times with the setting at `2`. This gave me an average runtime across three trial runs (to receive and serialize all JSON from all 1k async web requests) of 42.18 seconds.

Increaing `maxconnection` to 10 (up from 2) in the app.config gave me increased performance of an average runtime of 24.25 seconds across three trial runs (1.7x).
Edited by Laurabelle on 10/17/2012 8:41 AM PDT
Reply Quote
I just want to comment on Blizzard's excellent response times, specifically the low latency of the API.
Reply Quote
90 Human Paladin
14880
I tweaked the parallelism on my asynchronous GET requests to 1000 simultaneous maximum. Was able to pull item ID's 1 through 100000, serialize the JSON, and write to disk in about a minute. Another 3 seconds to load infile them to MySQL, so round about 1-2 minutes to build the cache.

At time of writing however, inside the domain {1-10000} there exists only a set of 59737 successful (200 response). So roughly 40% of my requests were 404's.

A potential workaround to not creating a JSON composite of ALL items, could instead perhaps create a new JSON that contains an array (list) of the "active" item ID's. In this way it could be consumed and only requests for expected active item ID's be made.

This would save a lot of unnecessary requests, as it appears at current item ID's are not reused and there are quite a bunch of holes.
Edited by Laurabelle on 10/22/2012 11:10 AM PDT
Reply Quote
90 Night Elf Hunter
5920
Are you willing to share the downloaded data? I have it on my list go update my database, but I would rather take advantage of a JSON dump somebody else has shared to save time and quota usage...

What I did was upload it to a google document site and shared it from there. It worked pretty good... especially since there seems to be a low interest in actually downloading it :)

Thanks!
Reply Quote
90 Human Paladin
14880
Zucanthor, I'm in the process of writing a tutorial on how I accomplished this. I will post a link to it when it is ready. I'm currently not in a position to share a composite JSON, when you formed it as a total composite did you just make a single object with one property that holds an array that holds each (valid or invalid so that key => item id) item JSON object?

Anyways, this tutorial is still very much a WIP but I posted a rough draft here: http://www.strategytraderhelp.com/wow/item/

*EDIT: I just updated the link with more information. I'll add further explanations when I have the time, but the heart of the content matter is there.
Edited by Laurabelle on 10/23/2012 8:37 AM PDT
Reply Quote
90 Night Elf Hunter
5920
On my system, I stored all the individual items as separate files, from file named "25" (the first item) up to file numbered "91016" (which is the highest numbered item I could find). The composite file is a concatenation of all those files (sorted numerically) with a header and a footer added to the composite file. I also do some indentation and ensure that all items (except for the last item) ends in a comma.

The format of the final composite file looks like the following:

{
"items" : [
{"id":25,"description":"","name": ... },
{"id":35,"description":"","name": ... },
{"id":36,"description":"","name": ... },
....
{"id":91015,"description":"Teaches you how to summon this mount.", ... },
{"id":91016,"description":"Teaches you how to summon this mount.", ... }
]
}

I am using Ubuntu and just wrote up a quick shell script to create the composite file from all the individual item files. Here is the script:

#!/bin/sh
(
cd itemdata_5.0/
printf '{\n "items" : [\n '
ls -v | xargs cat | sed 's/}{"id"/},\n {"id"/g'
printf '\n ]\n}\n'
) > itemdata_5.0.json

Since you are using windows, you will need to write up your own program to do the conversion. The important thing is making sure that the resulting output is a valid JSON file. I have confirmed that the JSON perl modules can parse the composite file without any issue.

By the way, what is the highest numbered item that you have found?

I also don't mind hosting the newly created composite file you come up with if you can get it into my hands. I just think that if somebody has already downloaded them all, it would save everyone else on bandwidth and quota usage to take advantage of it.

Good work on what you have done so far!
Edited by Zucanthor on 10/23/2012 12:45 PM PDT
Reply Quote
90 Human Paladin
14880
Zucanthor, I only tried the first 100,000 Item IDs.

Of that, I found 59,737 total that gave me a 200 (success) response code. I verified that all the others gave me a 404 (not found) response (on rare occasions I get a 503).

Of that set, the highest Item ID in-use was 91016. Which is an item named `Reins of the Great Red Dragon Turtle`.

Some back analysis from that ID shows a lot of different turtles. I will try going higher than 100k item ID's and see if there is perhaps just a large gap. As far as I know the API does not offer us a `MAX_ID_IN_USE`.

91016 Reins of the Great Red Dragon Turtle
91015 Reins of the Great Purple Dragon Turtle
91014 Reins of the Great Brown Dragon Turtle
91013 Reins of the Great Blue Dragon Turtle
91012 Reins of the Great Green Dragon Turtle
91011 Reins of the Great Black Dragon Turtle
91010 Reins of the Great Red Dragon Turtle
91009 Reins of the Blue Dragon Turtle
91008 Reins of the Black Dragon Turtle
91007 Reins of the Red Dragon Turtle
91006 Reins of the Purple Dragon Turtle
...


I see from your shell script and example that the the composite is as I had expected. I agree feeding the composite to a parser for validation would be a good idea.

*Edit* I just finished a run of the first 200k, and yea 91016 still remains on top.
Edited by Laurabelle on 10/23/2012 1:22 PM PDT
Reply Quote
71 Human Warlock
7150

I was working on serializing some JSON from the item API to build an item cache based on a few of the item properties in the JSON that I cherry picked to my liking.



What do you mean by serializing some JSON? I always have thought of JSON as a serialization format.


Anyways, off-hand I've no way to ascertain the upper limit on item id's, so I just assume it to be 100k.


Yeah, I can't find a way to get max ids for items or spells or anything either.


So, I fire off 100k asynchronous tasks to perform the GET requests on every item id URL (1 through 100k). Each task then upon completion updates my db with the necessary queries (queries are pooled on simultaneous connections). Assuming 200 or 404 response, if 503 we keep trying, task bails on any 500.

Honestly, that, or even the 1k you mention later sounds just evil. Sounds like a DOS attack almost. Can't you get most of the same benefits with keep alives, or does blizzard not support them?
Reply Quote
71 Human Warlock
7150


I am using Ubuntu and just wrote up a quick shell script to create the composite file from all the individual item files. Here is the script:

#!/bin/sh
(
cd itemdata_5.0/
printf '{\n "items" : [\n '
ls -v | xargs cat | sed 's/}{"id"/},\n {"id"/g'
printf '\n ]\n}\n'
) > itemdata_5.0.json


I think the first line should be '#!/bin/bash' so it can work even on systems that dont use
bash as the default shell and use the bourne shell like it was originally.


Since you are using windows, you will need to write up your own program to do the conversion. The important thing is making sure that the resulting output is a valid JSON file. I have confirmed that the JSON perl modules can parse the composite file without any issue.


Another option on windows would be to use cygwin or uwin or like that gives you a unix shell
environment.
Edited by Twobit on 11/27/2012 3:28 PM PST
Reply Quote

Please report any Code of Conduct violations, including:

Threats of violence. We take these seriously and will alert the proper authorities.

Posts containing personal information about other players. This includes physical addresses, e-mail addresses, phone numbers, and inappropriate photos and/or videos.

Harassing or discriminatory language. This will not be tolerated.

Forums Code of Conduct

Report Post # written by

Reason
Explain (256 characters max)
Submit Cancel

Reported!

[Close]