Historical Data

85 Night Elf Druid
2790
Would it be possible to get some historical auction house market data? I used to collect it, but I stopped for a few months. Now I'm back to collecting data every hour when new data comes out.

As this strategy takes a long time to get a decent amount of data to analyze (1-2 weeks), would it be possible to perhaps get raw historical data? Like for the past month or something?

That would be great.

Thanks,
Brandon
Reply Quote
MVP - Battle.net APIs
100 Draenei Paladin
14275
i dont think they would do this .... you best bet is to start storing the raw files and make a db of every thing ..
Reply Quote
85 Night Elf Druid
2790
Yeh, I'm logging everything right now. I have about 28 hours logged so far... once I get about 200 I'll have decent data. It'd just be extremely nice if they were able to give previous archives, and I'm sure they have them somewhere.
Reply Quote
Blizzard Employee
02/20/2012 09:52 PMPosted by Gamozo
Yeh, I'm logging everything right now. I have about 28 hours logged so far... once I get about 200 I'll have decent data. It'd just be extremely nice if they were able to give previous archives, and I'm sure they have them somewhere.


We don't have any plans on providing storage or access to historical AH data. If this is important to you, I suggest you take the route of downloading updates and uploading them to S3 (or the like) for processing and archival.
Reply Quote
85 Night Elf Druid
2790
Sounds good.

Currently I have my program check for a different lastupdated every minute, then downloads when there's a new snapshot (every hour). Is this an ok technique? I just like to make server use as little as possible.

I'm currently capping one server only right now, but I'm interested in databasing and releasing all of it to public domain, not sure if that would be ok usage. (numservers * 60 * 24) hits per day, and (numservers * 24) ah downloads per day. Currently I only care about the 226 US servers, so 325,440 hits per day and 5,424 downloads per day. I can decrease the hits per day by only checking every 30 minutes or so (or a bit of logic to get as close to checking every hour without missing data).

(using gzip obviously for bandwidth)

Reran the numbers properly. On a current AH pull of everything, it's an average of 350KB compressed per server. This means 79.1MB per pull, 24 times a day, so 1.8GB or 21.97KB/s consistent load.

Do these numbers look too high?

-Brandon
Edited by Gamozo on 2/21/2012 12:03 PM PST
Reply Quote
85 Draenei Priest
10130
Use http://theunderminejournal.com/

It keeps historical data.
Reply Quote
85 Night Elf Druid
2790
That site is nice, but not my goal. I have my own rendering and charting software, it's a data issue on my end. But I'm getting all I need now... just a matter of time before I have a few weeks of data. I might get a server to do this, so I'd eventually have years of data.
Reply Quote
100 Undead Mage
16400
Checking every minute seems excessive when the data is only updated hourly.
Reply Quote
Gamozo, I am very interested in your system for collecting hourly Auction House data. I am currently working with a research group on the economics of World of Warcraft and would be very interested in creating a system like yours. If you are interested in assisting please let me know.
Reply Quote

Please report any Code of Conduct violations, including:

Threats of violence. We take these seriously and will alert the proper authorities.

Posts containing personal information about other players. This includes physical addresses, e-mail addresses, phone numbers, and inappropriate photos and/or videos.

Harassing or discriminatory language. This will not be tolerated.

Forums Code of Conduct

Report Post # written by

Reason
Explain (256 characters max)

Reported!

[Close]