Yup. Unless the pin traders are coming up and harassing you, or have spread outside of their designated area, or whatever, why is it such a problem to ignore them if you're not interested?
The bottom line is that this is a hobby Disney has established and encouraged. The pin traders aren't invaders; they're guests of DLR and they've been encouraged to be there. And some really enjoy trading with them. If you don't want to participate just walk away.