|
Would it be possible to get some historical auction house market data? I used to collect it, but I stopped for a few months. Now I'm back to collecting data every hour when new data comes out.
As this strategy takes a long time to get a decent amount of data to analyze (1-2 weeks), would it be possible to perhaps get raw historical data? Like for the past month or something? That would be great. Thanks, Brandon |
|
i dont think they would do this .... you best bet is to start storing the raw files and make a db of every thing ..
|
|
|
Yeh, I'm logging everything right now. I have about 28 hours logged so far... once I get about 200 I'll have decent data. It'd just be extremely nice if they were able to give previous archives, and I'm sure they have them somewhere.
|
Web & Mobile Team
|
We don't have any plans on providing storage or access to historical AH data. If this is important to you, I suggest you take the route of downloading updates and uploading them to S3 (or the like) for processing and archival. |
#4
2/21/2012
|
|
|
Edited by Gamozo on 2/21/12 12:03 PM (PST)
Sounds good.
Currently I have my program check for a different lastupdated every minute, then downloads when there's a new snapshot (every hour). Is this an ok technique? I just like to make server use as little as possible. I'm currently capping one server only right now, but I'm interested in databasing and releasing all of it to public domain, not sure if that would be ok usage. (numservers * 60 * 24) hits per day, and (numservers * 24) ah downloads per day. Currently I only care about the 226 US servers, so 325,440 hits per day and 5,424 downloads per day. I can decrease the hits per day by only checking every 30 minutes or so (or a bit of logic to get as close to checking every hour without missing data). (using gzip obviously for bandwidth) Reran the numbers properly. On a current AH pull of everything, it's an average of 350KB compressed per server. This means 79.1MB per pull, 24 times a day, so 1.8GB or 21.97KB/s consistent load. Do these numbers look too high? -Brandon |
|
Use http://theunderminejournal.com/
It keeps historical data. |
|
|
That site is nice, but not my goal. I have my own rendering and charting software, it's a data issue on my end. But I'm getting all I need now... just a matter of time before I have a few weeks of data. I might get a server to do this, so I'd eventually have years of data.
|
|
Checking every minute seems excessive when the data is only updated hourly.
|
