Single large item cache download?

90 Orc Warlock
11235
Are there any plans for allowing us to download a single large file containing info for most (if not all) items that can be fetched singly via the usual API?

The AH dump on my realm generally contains 20k-30k entries, which are made up of ~3k unique items. Obviously, hitting the single item query API 3000 times is not going to be a valid option for non-registered API users like me, and even if 3000 retrievals were done by a registered application, I can't imagine you'd want anyone hitting your server(s) 3000 times in less than a couple minutes to retrieve and cache item data (even if it were only a one-time sc%#!@).
Reply Quote
100 Night Elf Hunter
12550
herd of hit cacheing? when my script polls a item it first checks internal if nto pulls and stores ..
but i dont see them giving us a 70k plus dump that would be massive...
Reply Quote
90 Orc Warlock
11235
Well if you had read the stickies, nonregistered API users are limited to 3,000 queries per day. And also, as mentioned before, one single large request is infinitely less demanding than thousands of single requests. Also, if again you had read my post, my realm's AH dump is sometimes 40,000 items. It's ~3 megs and it takes 1 request and 2 seconds to download. Offering up 70,000 items in one large single download that people would only logically need to download once per patch is a negligible step upward from the perhaps dozens of AH downloads one person might request per day (frequency depending on how often the AH dump is refreshed on Blizzard's end). So in your example, citing that 70k items would be 'too massive' is unfounded, when an AH dump is only half that size when going by item count.

If I go by your example, I would need over 23 days to obtain a list of all items, with a total of 70,000 separate web server requests (pretending to the extreme that I will need every single item's detail at least once). On the flip side, I could make ONE request and obtain all the data in about 12 seconds. My realm's AH dump is just under 4 MB, if transferred via gzip encoding (which Blizzard has indicated they'll be doing) this size is reduced by about 75% in my tests, on average. If we allow for a average of 1 kilobyte of data per item (the example on Blizzard's API documentation shows an item reply with a size of 917 bytes), we have an uncompressed file of about 68 MB. If that is gzip encoded, Blizzard ends up transferring approximately 17 MB. This is a giggle's worth of a download for any hosting provider, which most API consumers are probably coding for (aka website goodies hosted on professional web server providers), and which I assume you are also doing, since you mention scripts. It's definitely barely even a blip as far as Blizzard is concerned. It's even a non-issue for nearly all home users with any sort of broadband connection.

Take it from a web server admin, one large download (if we can consider 18 MB 'large') is much much much more preferable to several thousand automated, rapid-fire requests. Even if they are paced/throttled by the requester to a decent amount per minute, the processing and connection overhead for each single request far outweighs the resources used to transfer one large file through one request.

Finally, of course I cache. You miss the point, though. Try again, now that I have attempted to expand your knowledge on some of the finer details of resource and data distribution.
Edited by Lykaon on 8/5/2011 10:50 PM PDT
Reply Quote
100 Night Elf Hunter
12550
I understand what your saying i would love to have a total database of all items as well and yes 18mb is peanuts but they are also considering size of data requested too. and yes im developing a like profiles system for wowroster.net we are trying to get away from data from game because of the amount of issues people are having with addons.

Your point was not lost to me and as for requests go in 2 hours my site made 54549 requests just fixing my update queue.. that 1 guild list and 19x per member (run item api tho its not online) id like to see a a large file or passing a list of items info you would like in the header

What we Really need to do as a community is get all the dev's to make a site and post item dumps when we have them then any dev can dl them and update there database this would help on requests as well

I would also like to apologize if i seem obnoxious to you this was not my intent i was merely trying to add some humor in my response
Reply Quote
90 Orc Warlock
11235
Sorry about the rant.

Assuming Blizzard doesn't provide something like this, I might be willing to spearhead a community effort to do it. I've got the dedicated server and programming capability to do it, too. Not some piece of junk shared hosting website, but a completely dedicated server in a datacenter. :)
Edited by Lykaon on 8/5/2011 10:59 PM PDT
Reply Quote
100 Night Elf Hunter
12550
nice... we use to ahve a dedicated server .. then people stopped using roster .. we have a good data storage table in our system.. i also have scripts that phase the item api info into tooltips adding localised language text

this is what the comunity should be doing sharing and making it better for everyone
Reply Quote
Web & Mobile Team
08/05/2011 07:54 PMPosted by Lykaon
Are there any plans for allowing us to download a single large file containing info for most (if not all) items that can be fetched singly via the usual API?


No.
Reply Quote
90 Orc Warlock
11235
Okay, how about adding one additional line to the auction house data so we can at least have the name of the AH item instead of needing to worry about performing hundreds of additional requests just to get the item name? :) Caching the details is still, of course, the preferred option, but if we're writing an AH related app/tool, the previous request would help immensely on both ends.
Edited by Lykaon on 8/6/2011 11:04 AM PDT
Reply Quote
85 Orc Hunter
7715
Why can't you cache it now, and release it with your tool? Even if they add more items later, you still have the more significant ones now.

Or even, use an alternative site, such as MMO-Camp or Wowhead.
Reply Quote
90 Orc Warlock
11235
Because believe it or not, items change when hotfixes or patches are applied, and there is no way for a developer to be notified on a per item basis that it has been modified. You're left in the same spot, needing to query each and every item at least once per patch/hotfix. I can do that just fine, my intent was to reduce the load on their servers. Now I'm sure they're not hosting the api servers on an old pentium 2 on a DSL connection but reducing bandwidth and resource utilization is a primary goal for any professionally hosted solution whether they are doing it for profit or not.

Also, you apparently missed the part where I mentioned nonregistered users being limited to 3,000 queries per day.

Finally, using sites to query for data is rude unless you have consent from the operators. I'm not irresponsible enough to hammer wowhead or mmochampion's server with several thousands of requests without their permission.

Anyway, between Auctioneer and a custom addon I whipped up, I have exported all of the item data from my server's auction house and exported the item data to a file. It obviously is not going to contain all possible auctionable items but it is a start. This file currently contains only the info retrieved from the WoW API GetItemInfo, but I'm willing to export more information if requested.

If anyone else is interested in using or contributing to this item database, they can contact me on Blackhand (since iirc posting personal info on the forums is a no-no). It's currently available in JSON and XML formats.
Edited by Lykaon on 8/6/2011 3:33 PM PDT
Reply Quote
85 Orc Hunter
7715
My apologizes, hotfixes and patches slipped my mind... Need moarw sleep.
Reply Quote
85 Human Paladin
6775
08/06/2011 02:14 AMPosted by Straton
Are there any plans for allowing us to download a single large file containing info for most (if not all) items that can be fetched singly via the usual API?


No.


I strongly recommend considering changing your position on this, or at least allowing multiple items to be retrieved in a single query. Previous posters in this thread have spelled out exactly why having a bulk fetch solution would be preferable to having only a single item requested at a time.
Reply Quote
89 Tauren Druid
6065
nice... we use to ahve a dedicated server .. then people stopped using roster .. we have a good data storage table in our system.. i also have scripts that phase the item api info into tooltips adding localised language text

this is what the comunity should be doing sharing and making it better for everyone

I've used WoW Roster in the past and I've always wanted to ask: Why even grab the item info? You could use Wowhead's tooltips like everyone else.
Reply Quote
100 Night Elf Hunter
12550
wowhead tooltips dont show gem info see ulminia.wowroster.net?name=ulminia&server=Zangarmarsh

when wowhead applys gems they only give you the name not the stat and most days .. thay are slow as hell
Reply Quote
100 Undead Mage
10350
08/07/2011 01:03 AMPosted by Chalyss


No.


I strongly recommend considering changing your position on this, or at least allowing multiple items to be retrieved in a single query. Previous posters in this thread have spelled out exactly why having a bulk fetch solution would be preferable to having only a single item requested at a time.


I'm pretty sure the reasoning behind not allowing a large download is to prevent sc*%!%#@ of data. Also a large request takes more resources.

Yes there are benefits on the users end for only having to make one request but its probably not as beneficial on blizzards end
Reply Quote
100 Night Elf Hunter
12550
processing 70k+ items would take rufly 19-23 hours that average 6 sec a requesta nd processing time...
Reply Quote
89 Tauren Druid
6065
wowhead tooltips dont show gem info see ulminia.wowroster.net?name=ulminia&server=Zangarmarsh

when wowhead applys gems they only give you the name not the stat and most days .. thay are slow as hell

I'm pretty certain you could write a plugin to alter the tooltip as you see fit. If anything, you could just write your own wrapper for tooltips since all it's doing is making calls to their server and returning some JavaScript.

As far as speed, I'd deal with slow tooltips before I made 19 API calls per character just to stash item info. But that's just me.
Reply Quote
100 Night Elf Hunter
12550
true on both counts you can ajax the info f4rom there server and process it with php or what ever and store if needed
Reply Quote
85 Goblin Mage
11120
I'm storing everything in a database cache. If the item ID isn't in the database cache then I pull it from Wowhead.com and save it into the cache. I currently have 3,646 items and I just started building this cache 3 days ago. There will be a great number of items that will never be accesses thus no need to download. I'm guessing my cache will only contain about 10% of the total items in the game because so many are never even used anymore. There are close to 80,000 or more items.

Just cache everything like others have stated. You can cache them in a DB or in files within some cache directory (which isn't as fast).
Reply Quote

Please report any Code of Conduct violations, including:

Threats of violence. We take these seriously and will alert the proper authorities.

Posts containing personal information about other players. This includes physical addresses, e-mail addresses, phone numbers, and inappropriate photos and/or videos.

Harassing or discriminatory language. This will not be tolerated.

Forums Code of Conduct

Report Post # written by

Reason
Explain (256 characters max)
Submit Cancel

Reported!

[Close]