I think it would take a major 'shift' in society wherein everyone around would stand together and call these people out, or at least back up someone who did. I've seen situations where someone does call out some miscreant and the difference has been a loud chorus of "Yeahs!" and "That's rights!" from bystanders. As it stands, you take a big chance that you're going to be left twisting in the wind with no back up from bystanders or CMs (unless the CMs witnessed the incident).
I applaud your efforts to call these idiots out. I'll be standing right behind you. Like 20 feet behind you. LOL