[WoW] Easier way to collect player data?

API Discussion
Being that we have access to many of the wonderful available Blizzard APIs - there poses difficult hurdle for those who are trying to collect player data in an easy manner.

Say for instance you are a guild master and have over 250+ guild members. You also want to have a guild roster website that shows and tracks each guild member's data that is available from the Blizzard APIs. Sounds simple right? It should be? Umm, well maybe?

First you would have to gather the pertinent info about your guild. This is the simple part.

Next you have to parse the guild members API just to collect all the guild member's names (and some important data like: Spec, Role, Level , etc.)

Now you have this data and have displayed it on a website formatted in a roster form and it looks cool. Yet, there is other information that you would like to provide a website viewer without having them click all over the place and or go to other websites outside of your own page.

This "extra" info that you would like to display could be: Guild Member Join Date, Honorable Kills, Gear Item Level, Mounts/Pets/Toys Count, Secondary Specialization and a few more options of player data.

Here's the dilemma: Some of the "extra" player data is already available but it is located on separate APIs.

Thus an app/website designer would have to parse each and every guild member name and pull the respective character API, then pull that character's data. Well, for a small guild, this would be easy. But for much larger guilds, this is a time consuming process.

Example: My guild has 325 members. I would have to pull 325 individual Character APIs just to get the "HonorableKills" data from each player so I can display that field on my guild roster website. As well, I would have to do that same number of parsing names just to get the "AverageItemLevel Equipped" data and so on and so forth.

If you are actively pulling such data via a webpage call, cronjob or anything else, it is going to take some time just to pull, gather, populate, compare or update said data. Because for every name you have listed, it must pull each character API (and different field) to get some of the data you may need. So for just 2 fields to be displayed, that would require 650 API calls in my unique case needs.

So what I am asking is: Are there any more reliable & faster ways to pull Character API data (concerning more than 250+ calls) or could some of the Character APIs be simplified or condensed so that the number of calls can be reduced and not be so redundant for such little amount of information needed?
One workaround I am using is to have a database, and store the data into it once per day.

Then for the webpage I use the data from the database rather than call the api everytime the page is retrieved.
Obviously the more times you update the data to your database, the more accurate the data. But I don't think the data will change that much in 24 hours.

But yes I too would like the option to specify fields.

Edit: Btw where did you find member join date?
I'm really struggling here. It's easy to pull information about one character from the API. However as soon as you start messing with the entire guild, I find that the data is too nested or Im too stupid to make it work properly.
Peter has the right idea, you shouldn't be making your pages call the blizzard api directly.

1. They are too slow.
2. The data is inconsistent, ALL THE TIME. So your database can handle that logic.

So, periodically throughout the day, you should be pulling your data from blizzard, yes it's going to be a lot and yes the bandwidth is going to add up. So you will have to decide the appropriate balance and ask yourself, how often do you need to update?

Keep in mind there are websites that pull the entire pvp ladder for every bracket every 30 seconds, which is well over 24 thousand records, so this is small in comparison. Either way, it's going to require some blood sweat and tears to get your project off the ground.
It is understood and good practice to pull the data from the APIs into a database.

I am currently doing just that.

What I am asking is if there is a much more "consolidated" API for pull such data.

Right now I am having to pull data from 4-5 different APIs just for the data of my guild members. Each data pull is done every 3 hours but as I stated; I have to pull over 300 characters each time for small bits of data. There is data on some APIs that can be used together but also there is data that can only be found on certain APIs alone. I can proved the codes for the below call examples:

API data pull #1 = Guild Members
API data pull #2 = Character Thumbnails/Images
API data pull #3 = Character Lastmodified
API data pull #4 = Character AverageItemLevelEquipped
API data pull #5 = Character Honorable Kills
API data pull #6 = Character Reputation

If all this information was condensed into 1 Chracter API or Guild Member API -> it would be so easy to just pull that API, parse/collect the data we need instead of macking so many calls repeatedly.
I understand what you're saying now.

I just messed with the api a bit and found out you can do this:

https://us.api.battle.net/wow/character/zul'jin/bledfordays?
fields=stats&fields=pvp&locale=en_US&apikey=oopslol

this might save you a few calls!
I think you're supposed to csv the fields:

us.api.battle.net/wow/character/realm/name?fields=stats,pvp,items,achievements&apikey=qwertyuiop
Looks like that works too. I swear I tried that earlier and it didn't work for me, I must have mistyped something.
Is anyone doing their work in javascript or are you guys strictly PHP?
I'm .net and javascript. I use angular tho.
11/12/2014 01:34 PMPosted by Bledfordays
I'm .net and javascript. I use angular tho.


I'm using angular as well. I can make requests for single characters no problem. I can also make a request that gets the guild members dump. I'm running to trouble trying to save the characters to my db (mongo). This is pretty much a learning project for me.
Is your issue getting the JS to hit your .NET code? Or is the issue getting your .NET code to hit the database?
Thanks for the fields tip folks.

But DaPimpShield, how did you get member join date?
DaPimpShield, did the "fields" query parameter help address your issue or is the data you're looking to retrieve further spread out than that?
11/24/2014 04:55 PMPosted by Aurifex
DaPimpShield, did the "fields" query parameter help address your issue or is the data you're looking to retrieve further spread out than that?

Yes the combined "fields" parameters are doing just fine but I still have to pull each guild member's query to populate the data into the database.

Whereas with the Guild->Members API = I only have to pull that one query and then parse all the guild member's data from that.

I was simply asking if some of the data that is found in an "individuals" Character API could be included in a Guild->Members API so it would reduce/eliminate the need to pull each and every guild member's own personal APIs.
Ah, gotcha. I'm not aware of a way, but I can certainly kick that feedback over to the WoW API folks.
There definitely should be an easier way then to having to pull each individual character that is in a guild to grab the data you want from the characters that's missing from the guild information.

I doubt we can get a dump of all characters in a guild with every possible field, especially because some guilds have 999 members, but we should at least have the guild return the most basic information exactly identical to requesting a character without fields.

Right now there's fields that are missing from the guild members request:
- totalHonorableKills
With the methods I am using to collect every one of our guild member's API data; it's taking over 4-5 minutes to just pull 100 individual API files. Now mind you that I am calling the member API from a database list and also creating/saving the file to JSON format to my local server. Yet it shouldn't take 4-5 minutes to pull 100 individual API.
Here is the code I am using. It grabs the guild member names from the database, call each guild member's API and the creates/writes/save the API into a JSON file on my local hosted server. It works fine but sometimes there are guild member APIs that return NULL and in turn are recorded as zero byte file.
<?php
header("Content-type:text/html; charset=utf-8");

$time1 = microtime(true);

include "CONNECT_FILE";
$table = "DB_TABLE";
if (!$link) { die(); }
mysqli_set_charset($link,"utf8");

$sql_0 = mysqli_query($link,"SELECT * FROM $table");

include "API_KEY_FILE";
while($row = mysqli_fetch_array($sql_0)) {

$gr_ID = $row['gr_ID'];
$gr_PName = $row['gr_PName'];

$get_api = file_get_contents('https://'.$RegionName.'.api.battle.net/'.$GameName.'/character/'.$RealmName.'/'.$gr_PName.'?fields=items,mounts,pets,reputation&locale='.$LocaleName.'&apikey='.$APIkey.'');
$dec_api = json_decode($get_api,true);

$filename = 'api-'.$gr_ID.'.json';
$filename = iconv("UTF-8","ISO-8859-1",$filename);

define('DIRECTORY','/XXX/public_html/json');
file_put_contents(DIRECTORY.'/'.$filename.'',$get_api);

}

mysqli_free_result($sql_0);
mysqli_close($link);

$time2 = microtime(true);
echo "Script Execution Time: ".($time2-$time1)." seconds";
?>
I'm doing same thing as you. This is my request URL: http://eu.battle.net/api/wow/character/Thunderhorn/Kamenitxania?fields=guild,items,titles,talents,professions,audit

If response in not correct or not back in 1,5 second, I just query it again. These character request are parallel (16 threads) so I can get data from 200 guild members in 10 seconds.

Whole source: https://github.com/kamenitxan/luzanky-rainders/tree/master/src/cz/kamenitxan

Join the Conversation

Return to Forum