Most studies seem to indicate that the ear is relatively insenstive to phase variations between the different frequency components. This makes some sense if you think about how the ear works -- the inner ear converts directly to frequency domain before entering our nervous system, so the actual waveform shape is never directly sensed.
This is where the argument that flat frequency response correlates better with perceived sound quality than any of the phase/time coherence or group delay measurements.
My *guess* is that this may be more of an issue at the lowest frequencies, eg. we might be able to hear phase funnies at a few hundred hz because the delays are significant enough that our relatively slow nervous system can detect them. There is a debate about this in the subwoofer thread that Ajax posted... maybe the tiniest bit of validity there but IMO time coherence is not a big deal at higher frequencies.
Something else to discuss with Ian and Alan while we're sitting on the dock in September.
"Because of the tapered design of the cochlea, waveforms traveling down the basilar membrane peak in amplitude at differing spots along the way according to their frequency. Higher frequencies peak out at a shorter distance down the tube than lower frequencies. The hair cells at that peak point give us a sense of that particular frequency—it is thought that a single musical pitch is perceived by 10-12 hair cells. Due the tapered shape of the cochlea, the distance between pitches follows the same logarithmic distance as our perception of pitch i.e. the placement of octaves are equidistant. "
http://www.indiana.edu/~emusic/acoustics/ear.htm 
