Except, of course, that they really didn't. And those folks who say they don't listen to works by the Second Viennese School (Schoenberg, et al) hear quite a lot of music inspired by it -- and quite often.
Because while concert goers may have been slow to embrace the sonic possibilities of 12-tone music, many musicians and composers have -- especially film composers. They found the unsettling and vagueness of music freed from a tonal center ideal for suggesting menace, horror, mental instability, and similar emotions.
The clip below beautifully illustrates that. Michael Monroe noticed a very strong similarity between the suspenseful music cues used in the very mainstream "Andy Griffith Show" and Anton Webern's work. He substituted one of Webern's "Six Pieces for Orchestra, Op. 6," with the following result.
So consider giving the Second Viennese School a chance. It's really a musical language that's very familiar, even outside of a visual context.