Now, I don't think most diehard Disney fans fall into this category! I'm mostly talking about teenagers and families who look at Disneyland as a cheap thing to do on the weekend. It's basically something they think of as free, so they treat it no better than a public park or a city street.
Disney responds by trying to drive up attendance even further, because the annual passholders tend to spend very little in the park. So, the park becomes increasingly crowded and unpleasant, cast members are overwhelmed and overworked (leading to higher turnover), and attractions are pushed to their max limits.
If you did away with the annual pass, you might only really be impacting 700,000 or so people-- but the "high user," the ones who pay the least and visit the most.
For the first 35 years or so of its life, Disneyland was a DESTINATION, a special experience. You'd pay extra for it. Knott's Berry Farm or Magic Mountain was the weekend "amusement park," Disneyland was, well, Disneyland.
But when the AP concept really took hold, the whole thing started falling apart. It isn't sustainable, either financially or operationally, but I don't expect Disney to give up-- not easily.
In the meantime, I've always wondered what would happen if Disney LOWERED its admission price to say $45 for adults and $25 for children -- but offered absolutely no discounts of any kind, ever, except for employee/cast members and multi-day ticket purchases? What would happen if EVERYONE had to pay the same price, if there was no incentive to just "hang out" at Disneyland? Would attendance go down? Likely. Would per-capita spending go up? Also likely, because those in the park would be there as a special occasion, not just to kill a few hours on a Saturday evening.