The alliance reach our base before we can reach theirs, it's a very basic issue. So they have a 4 cap before we do it's just the way the terrain is designed. The core issue has nothing to do with back capping or defending, it's the terrain.
Sure you'll win more games if you back cap and defend, but there will always be those games that slip through your fingers by one tower or by a few seconds. Those are the result of alliance having a terrain advantage, not to mention the majority of alliance having EMFH human racial for a free pvp trinket while the horde have to buy theirs (seriously who buys a pvp trinket before hitting level cap).
But keep blaming the map for your refusal to recap any towers. IB GY is the best chokepoint on the map, but most Horde are like you and have no clue how to use it.
All of my AV achievements and the majority of my wins were Horde side.
A little defense goes a long way. Recap IBT/TP and you will probably delay the Alliance long enough to win.