Regarding why people feel the AH is a design mistake - the drop rates, that is, the standardized Diablo 3 drop rates (overall) are based on an auction house trading system instead of the trade/barter system in Diablo 2. As such, just finding loot that is an upgrade for you is a real problem for people, and I'm talking even leveling up. The entire drop rate mechanism is designed to drive you into the AH at some point, which will vary depending on the player's tenacity/patience.
The problem here is not one of loot quality, but sufficient quantity, especially during leveling up, to find upgrades that actually feel like (and are) upgrades. Nephalem Valor is a royally misguided crutch, and only works for players at max level, which is the exact opposite of the way it needs to work (assuming we maintain D3's abysmal loot distribution scenario as is). In Diablo 2, rares were relatively...rare, and uniques, while rare were still found a whole freaking lot more than in Diablo 3.
During my level up process on my first Monk, I didn't find a single legendary until LV 53 in Hell. And of the few rares that did drop (remember, no NV until LV 60), only a single one was even an upgrade at the time.
There is also the problem of item relevancy even during leveling up. That is, most rares and legendaries are still ten levels past their relevancy at the location in the game where they drop. This is a big problem. Because of this, unless you buy the (also misguided) "Reduces Level Requirement by XX" items from the AH or farm them on a max level character of your own, you ultimately feel weaker as you level rather than stronger. That's one of the biggest letdowns in the entire game. Items that drop two full acts past where they're needed and useful for are utter garbage. This major problem is the main factor driving people to the AH system, because natural progression during leveling up happens so rarely that it just feels awful, and the sad part is that this is how the game was intentionally designed.
It would seem that our experiences diverge radically on this point. I played hardcore almost exclusively, which meant that I had to take a LOT of characters from 1 to 60. After a while I realized it was far more fun to do it without the auction house, and without using any gear stashed from previous characters. Yet in the many times I leveled to 60, not once did I ever have a problem finding appropriate loot myself. Despite playing in HC (where it's obviously much more important to have gear appropriate for what you're tackling) I did not have to rerun any areas to farm up or outlevel the content. Furthermore, as I mentioned with my softcore barbarian, I was able to get to the point of efficiently farming A3 inferno after a grand total of 20 hours without using the AH. I simply couldn't disagree more about quality loot dropping, so long as we define quality in terms of the ability for that loot to help you clear content.
In fact, I had far more difficulty self-finding appropriate loot in D2 than in D3. It was very challenging (albeit fun) getting through A1 hell on a hardcore character using only self-found loot without first farming nightmare bosses to a great extent in D2.
One very big issue with this game that could actually help remedy the issue of caps (and which I haven't heard any discussion of myself) is the issue of which affixes roll on which types of items.
Currently, items basically serve to fill up a certain slot with a good roll of the same things. Attack speed, crit, crit dmg, etc.
Well, why should I be looking for a chest armor that increases my attack speed? I have one, but why does even an set or unique chest piece improve my damage at all? It's a piece of armor. If its properties focused on different kinds of protection exclusively (with the exception of giving primary stats, perhaps) then it would actually feel like it was chosen to be a piece of armor and not a pseudo-weapon.
Same thing for pants. Why do my Inna's pants improve my attack speed and critical hit? (Granted they are the only ones that can and it's only a 1% crit chance, but you can bet that 1% crit chance combined with attack speed makes them a very popular item among many classes). They're pants. Even unique pants should focus on movement/armor.
The way I see it, we should have an itemization system that treats items not as placeholders for affixes but as actual items that do what they appear to. Here's a possible example schema:
You've articulated the proposed changes clearly, but you haven't explained exactly how this change would address the issues affecting itemization. If I'm not mistaken, you're suggesting this as an alternative to caps/sharp diminishing returns. What you're proposing would reduce the pool of potential affixes on each item slot. Presumably you have in mind the implementation of new affixes as well, albeit ones restricted to the appropriate item types in your layout. But it seems to me that there's still the fundamental issue of any new affixes having to compete with existing ones. e.g. any weapon will have to compete with Skorn's massive crit damage. My worry about your proposal is that it will put significant constraints on the design of new items (no offensive affixes allowed in defensive slots, no defensive affixes on weapons, etc.) without offering a clear improvement to itemization.