From a physical standpoint, Fastpass "reserves" your place in 1 line while you wait in another line. By that very definition it is making the line longer because the Fastpass attraction must keep a portion of it's line underutilized specifically because it has to meet the Fastpass requirement of below 20 minutes.
From a psychological standpoint Fastpass rewards people for waiting longer. People are trying to kill time in order to meet their Fastpass window. If a line takes 2hrs... great... they have 2 attractions they're "getting" out of that wait. Remove the reward, and the line becomes merit based. People will only wait as long as they feel the attraction is "worth it". Most people won't wait 2 to 3 hours unless they have a Fastpass in hand while they're waiting.