Single large item cache download?

85 Goblin Warrior
710
Here's all the items (no tooltips, but some interesting bits) I've seen from crawling the auction house:

http://theunderminejournal.com/itemlist.xml

That's from my service that can, um, watch the AH for items, then email you when they appear. Among doing other things. But don't let that discourage you, it's an interesting exercise nonetheless.


Cheers, and yes it is an interesting exercise. I'm yet to do much with all the guild/character data I'm pulling in, other than fill disk space and maintain a guild roster, but there's a ton of stuff I'm looking forward to tracking over time :-)
Reply Quote
55,000+ whatever items divided by 3,000 requests per day is still 2 1/2 weeks of doing nothing but item downloads. There needs to be a better way.
Reply Quote
100 Night Elf Hunter
13825
for most people having all item data is undeeded but there are things you can do some of then not the best idea others so so but if you cache 1-2k items a day ... your good remember ah dumps have many of the same items over and over agian ... why cache the same item info 30+ times?
Reply Quote
90 Tauren Druid
11235
There are approximately 9,500 unique items in the AH that I've aggregated over the last week, if anyone is interested, I'll post up links to the files. My app backend keeps them updated every half hour.

The compressed file weigh in at a scary 671 kilobytes, uncompressed the XML file is ~12MB and the JSON is ~6MB.
Edited by Lykaan on 8/18/2011 1:28 PM PDT
Reply Quote
1 Human Paladin
0
Hey guys!
I made a thread on the EU forums about pretty much the same topic, and there's actually someone who downloaded all the item files and put them up for grabs :)

You can find my thread here: http://eu.battle.net/wow/en/forum/topic/2423036135#1
And the download link in post #8.

Hope you can use it :)

//Lokos
Reply Quote
2 Blood Elf Rogue
0
I linked to that myself already in this forum :P
http://us.battle.net/wow/en/forum/topic/2973395906#3
Reply Quote
1 Human Paladin
0
Hah, alright :) Well can't hurt to have it multiple places... Makes it easier to find :P
Reply Quote
Hi, I know this is last year's thread but I tripped over it whilst researching something I wanted to do against the WOW Api so figured I'd reply here.

I have my own item cache file with around 60,000 items in it, which is in XML format.

The way I did this was to download ItemDataCache addon from curse.com and then look at the item LUA file in there. It has a list of valid ItemIDs with sell prices. All I needed was the valid itemIDs so I parsed this to get a list of ItemIDs.

I am actually using C# .Net with the WOWSharp library. The library allows you get to get an auction dump, but the items are only IDs, no item names or other info. There is a method to get an item by ID, but that item object isn't serializable, and I wanted to serialize my own item cache onto disk. There's I made my own GameItem class with most (at least the serializable) properties and this can be created from a WOWSharp item.

So I used a SerializableDictionary (a third party custom class since standard Dictionary in .Net won't serialize) with the ItemID as the key and my GameItem object as the value. I have one routine which will generate the entire XML file (takes a good 1.5 hours since it has to do 60,000 server calls to get the item info 1 by 1). But once it's done it uses this, and probably you'd only want to regenerate the file occasionally, maybe once per month or so.

When my apps loads up I then deserialize this xml file into an object so I have all 60,000 items available in memory.

This allows me to get an auction dump and I can very quickly get any item's full information by ID in memory without having to make a server call. When I have to string search 50,000 auctions this is literally 1000 times faster than live-lookup of item name on each item in the auction dump.

My program does things like automatically checking if anyone has undercut any of my auctions (a lot easier than manually price checking 200+ auctions), or being able to search Horde, Alliance and Neutral AH for mulitple words or phrases all in one click - handy when looking out for several different types of transmog gear.

Where there's a will, there's a way :)

Now if only I can find a way for Blizzard to fix the Sha of Anger loot bug and generate the 4 roll chances I have missed so far because of it! ;p
Edited by Maplebunny on 11/2/2012 9:42 AM PDT
Reply Quote
90 Human Paladin
14880
I have a small tutorial regarding C# and item api:

http://www.strategytraderhelp.com/wow/item/

Basically, I can make 100k requests, serialize the JSON, and store to flat file in about a minute or so. With the flat file you can do whatever you wish, in this I load it to MySQL. In the tutorial I don't pull every item attribute, but it can easily be extended to do so.

08/11/2011 10:29 PMPosted by Roncli
55,000+ whatever items divided by 3,000 requests per day is still 2 1/2 weeks of doing nothing but item downloads. There needs to be a better way.

Think of it as 3k credits per day, not 3k requests per day. Each request has a different credit value associated with it. It appears an item request is only a small fraction of a credit. Credits and actual GET requests are not one-to-one.

Remote web servers can have a large amount of latency and while waiting we should be trying to put load on the network as much as possible so that we can get our requests done as quickly as possible. The .NET framework imposes a limit of 2 simultaneous connections on a same host. However, Blizzard's servers can easily handle much more, so this setting should be increased in the app.config.

11/02/2012 09:39 AMPosted by Maplebunny
(takes a good 1.5 hours since it has to do 60,000 server calls to get the item info 1 by 1).

Judging by this, you have not increased the connection limit and you're not making the requests asynchronously. It should be much faster than this.
Edited by Laurabelle on 11/5/2012 7:12 AM PST
Reply Quote
3 Draenei Shaman
0
Laurabelle i've been trying to compile your c# and it seems that System.Json isn't comming up in the NUget package anymore. only ones its returning is
NETFx JsonSerializer as you can see from the NUget website https://nuget.org/packages?q=system.json
Reply Quote
90 Human Paladin
14880
Talonos -- strange, I can't say for why it is not showing up in the search anymore. It is still there, though: http://nuget.org/packages/System.Json

Json.Net is also a popular one. Also, the "old" way of setting up the data contracts is still valid as well.

If you do tools --> Library Package Manager --> Package Manager Console

and use this on the console:

Install-Package System.Json -Version 4.0.20126.16343

it should put it in for you. Additionally, I added in this info on that page. Let me know if you run into any other issues. Thanks!
Edited by Laurabelle on 11/6/2012 6:13 AM PST
Reply Quote

Judging by this, you have not increased the connection limit and you're not making the requests asynchronously. It should be much faster than this.


Hi - thanks, yeah I am doing nothing asynchronously and haven't increased the connection, didn't know I could tbh, but then I wrote this is one evening from a standing start so was just trying to walk before I ran :)

But then honestly how often are new items added or changed? I might run the item cache update maybe every couple of weeks or so, but at such a low frequency
Reply Quote
90 Human Paladin
14880
11/06/2012 08:12 AMPosted by Maplebunny
But then honestly how often are new items added or changed? I might run the item cache update maybe every couple of weeks or so, but at such a low frequency


In this specific case, you are correct. However, it is not that huge a leap to add in this functionality since the .NET framework provides it relatively painlessly. I mean, call it an academic win or what have you, but making your program operate 75x faster is never a bad thing in my book =p
Reply Quote
71 Human Warlock
7150


The way I did this was to download ItemDataCache addon from curse.com and then look at the item LUA file in there. It has a list of valid ItemIDs with sell prices. All I needed was the valid itemIDs so I parsed this to get a list of ItemIDs.



As the updater of that version of ItemDataCache I got a chuckle out of this. To make that
item list I in turn striped down a data dump someone made of the items from the api.

https://www.box.com/s/eyvlqj3z4bwxufbcxqsw

I do agree with the original poster that it would be nice to have Blizzard provide the dump rather then having to loop through the item numbers.
Reply Quote

Please report any Code of Conduct violations, including:

Threats of violence. We take these seriously and will alert the proper authorities.

Posts containing personal information about other players. This includes physical addresses, e-mail addresses, phone numbers, and inappropriate photos and/or videos.

Harassing or discriminatory language. This will not be tolerated.

Forums Code of Conduct

Report Post # written by

Reason
Explain (256 characters max)

Reported!

[Close]