Topic HTML Datamining and You
Osundir
Osundir
Web & Mobile Team
Edited by Osundir on 8/8/11 6:18 PM (PDT)
Now that the API is live, we want to make you all aware that HTML datamining (via us.battle.net/wow or any of the other regional wow sites) is not acceptable. Within the next 7 days, we will start to blacklist users that are making excessive requests to our regular community sites.

If you are currently in the process of porting over to the API and have not requested a key yet, please do so. We do not want to interrupt service for community sites that are following that path.

Please email api-support@blizzard.com with specific questions/concerns, or reply to this thread with any general comments.

Thank you for your cooperation!
Tericho
Shattered Hand
Tericho
85 Undead Mage
4765
I hope you gave the developers enough notice, could be a lot of work to turn these sites that rely 100% on HTML parsing into API driven apps.
Mattick
Lightninghoof
Mattick
85 Gnome Mage
2270
I hope you gave the developers enough notice


This IS the notice ...
Sabatieni
Winterhoof
Sabatieni
90 Troll Warrior
10090
Will this blacklist sites similar to Zardoz's old Armory Datamining website?
Sabatieni
Winterhoof
Sabatieni
90 Troll Warrior
10090
Here's the relevant link for those interested:
http://armorydatamining.appspot.com/

He compiled population data to determine class, spec, and race frequency amongst players, among numerous other data points.
Chaud
Gurubashi
Chaud
90 Undead Mage
12675
Does this include sc!%%@!@ the forums?
Ulminia
Zangarmarsh
Ulminia
90 Night Elf Hunter
10640
08/08/2011 07:29 PMPosted by Chaud
Does this include sc!%%@!@ the forums?


if its a battle.net site then yes its a no,no
Osundir
Osundir
Web & Mobile Team
@Tericho: Consider this the notice. We've had the API 'out there' for 2-3 months now, so there really isn't an excuse to have not STARTED porting your code over by now.

@Sabatieni: I believe he was mining the old Armory data that we decommissioned last week, but if he switched to the new site, its possible.

@Chaud: This is really targeted at data we support on the API (Character Profiles, Guild Profiles, etc).

We won't be targeting Forum data miners initially, but there is an off-chance some may get caught up in the API miner sweep, so I'd suggest contacting us if that might be the case.
Arkeus
Lightninghoof
Arkeus
90 Dwarf Paladin
13650
If we were blacklisted from earlier on before the release of the API (strangely the same amount of requests I was making for awhile ended up enough to get my blacklisted a few months ago), is it possible to get unblacklisted so we can use the new API? I shot an email to api-support@blizzard.com a few weeks ago and never got a response.
Sabatieni
Winterhoof
Sabatieni
90 Troll Warrior
10090
08/08/2011 07:44 PMPosted by Osundir
@Sabatieni: I believe he was mining the old Armory data that we decommissioned last week, but if he switched to the new site, its possible.


Thanks Osundir, that's what I was wondering. I knew everything was "cool" back in the day when he was still maintaining the blog, but wasn't sure if someone could attempt what he did with this new policy.
Sabatieni
Winterhoof
Sabatieni
90 Troll Warrior
10090
@Chaud: This is really targeted at data we support on the API (Character Profiles, Guild Profiles, etc).

We won't be targeting Forum data miners initially, but there is an off-chance some may get caught up in the API miner sweep, so I'd suggest contacting us if that might be the case.


Do you think this might catch Blue trackers, like those hosted by MMO-Champion?
Cheevos
Uther
Cheevos
85 Goblin Warrior
710
08/08/2011 06:16 PMPosted by Osundir
Please email api-support@blizzard.com with specific questions/concerns, or reply to this thread with any general comments.


I've emailed in with my specific concern, but I'll repeat it here for community discussion (or blue deletion if it's offtopic). I sc!@#* the guild news feed to present on my guild forums.

I could recreate the achievements portion of it by munging through achievements history for all characters and guild, I expect, but that really also depends on having the achievements data resource up and running :-)
Delmortis
Jubei'Thos
Delmortis
85 Troll Priest
13100
What about for those things are are pretty essential for those of us that wish to view all the relevant character profile information but can't with the new API as it doesn't support pet talents yet?
Sixen
Black Dragonflight
Sixen
MVP
58 Blood Elf Death Knight
500
08/08/2011 07:44 PMPosted by Osundir
We won't be targeting Forum data miners initially, but there is an off-chance some may get caught up in the API miner sweep, so I'd suggest contacting us if that might be the case.

That in mind, some kind of Battle.net Forum API on top of the WoW Armory API would be huge...
________________________________________________
The Chat Gem Lives!
Fansite Administrator: DiabloFans, SC2Mapster, SC2Replayed
Osundir
Osundir
Web & Mobile Team
Just to be clarify further - unless you're making 1000s of requests a day, its unlikely you'll get caught up in this :) This is highly unlikely to apply to blue tracker sites (Assuming they are doing it sensibly), or personal guild sites.
Chaud
Gurubashi
Chaud
90 Undead Mage
12675
Only 10,000 - 20,000 a day!
Zeherah
Stormrage
Zeherah
90 Dwarf Hunter
14165
08/08/2011 09:26 PMPosted by Delmortis
What about for those things are are pretty essential for those of us that wish to view all the relevant character profile information but can't with the new API as it doesn't support pet talents yet?


I would hope they're not cracking down on data that's not fully supported yet. For now I've changed my script to get everything but pet talents through the API and then doing a second request for the pet specific data through the HTML. I don't think I'm usually doing more than 1k requests per region a day, so I would hope that's within the realm of acceptability.

I'm also still using the HTML version of the TW armory for character imports as the last I checked the character APIs there were not working properly (I hope they'll let us know when it gets fixed). Hopefully this isn't going to cause a problem either.
Chalyss
Rexxar
Chalyss
85 Human Paladin
6735
08/08/2011 09:47 PMPosted by Osundir
Just to be clarify further - unless you're making 1000s of requests a day, its unlikely you'll get caught up in this :) This is highly unlikely to apply to blue tracker sites (Assuming they are doing it sensibly), or personal guild sites.


Forum and activity feed APIs would still be huge and awesome. :)
Azxiana
Feathermoon
Azxiana
85 Night Elf Priest
4630
I believe it is perfectly reasonable to shut out those still HTML sc%%*#*!, but ONLY if all the data is available through the API.

I really hope that PvP information is available through character and team pages within this one week deadline.
Spellpower
Dreadmaul
Spellpower
85 Undead Mage
6690
Edited by Spellpower on 8/9/11 5:05 PM (PDT)
What qualifies as datamining?

One of my favorite sites, The Undermine Journal [http://theunderminejournal.com] uses the API. Could that be considered datamining?

Is that the type of thing that wont be allowed

Edit: I fail at english

Edit: I fail at english even more, its HTML datamining not API..... derp

Please report any Code of Conduct violations, including:

Threats of violence. We take these seriously and will alert the proper authorities.

Posts containing personal information about other players. This includes physical addresses, e-mail addresses, phone numbers, and inappropriate photos and/or videos.

Harassing or discriminatory language. This will not be tolerated.

Click here to view the Forums Code of Conduct.

Report Post # written by
Reason
Explain (256 characters max)

Reported!

[Close]