The 3.x client had the exact same limitations on VRAM caching as the 4.x client on the OS X side. It was the Windows side that increased cache allotment capability, and the LAA extension flag on Windows is only accessible via a (sanctioned by Microsoft) very specific method for which there is no equivalant in OS X.
I was pretty sure that the problem was system RAM, not VRAM. Otherwise, why would the problem have only started in the 4.x client?
The reason the problem started with the 4.x client is simple: 3.x's "ultra" settings were based on older OpenGL code and had lower thresholds than do our current Ultra settings. The current Ultra settings place a huge
strain on the asynchronous data offload functions in OS X (cache swapping to/from VRAM) because on our client, only textures in the field of view (FoV) are kept active while those no longer in FoV are dumped, and if you turn quickly, they must be reloaded into VRAM from their hardfile source (the MPQs). The PTR client lessens this effect to a modest degree from what I can see, but it is still very much prevelant, so I'm guessing the client is only allowing maybe 128 MB of texture cache at the most. Had Blizzard enabled PAE (Physical Address Extension) capability, normally "auto-on" in OS X built apps by default, the client could cache as much textures as VRAM could hold. In the case of Mac video cards, that's 1 GB (1024 MB), a far cry from the 64 MB Live limit, and still very much larger (8x) than the 128 MB limit I'm surmising the PTR client has.
Don't get me wrong - the PTR does nave noticeable gains in performance, but it's still nowhere near the fluidity of the 3.x client. One other thing to keep in mind: I used to run Ultra settings for nearly everything other than ground clutter in 3.x on a Radeon HD3870. I'm currently running a Radeon HD5870 in the same machine and can't do Ultra in the 4.x client, despite being two full generations separated from the previous card, faster VRAM, twice
the VRAM, and a more up to date OS version. That's pretty telling right there.
I know the devs are still working on this actively. I just wish they'd let us try a PAE enabled PTR client with an allowable texture cache of 1024 MB, if for no other purpose than to test whether or not that could fix this issue by itself. The Live client dies hard at 2 GB. There's nothing the user can do to extend that RAM table - on Ultra settings, you will reach 2 GB
in the new zones and the client will simply die on you. We really need access to PAE if we're going to ultimately solve this, since the tools available to us in a cramped 2 GB RAM limited environment are simply too lacking. You can only squeeze so much juice out of a tiny lemon before the pulp clogs up your glass. We need a bigger lemon from which to make our lemonade.