My problem with "ears alone", is that is very rarely the case. Unless you are able to conduct a proper double-blind listening test (you don't know which component you are listening to, and the person making the change has no possibility of communicating this to you either), your mind will often mislead and you'll "hear" the dollar signs or some other factor which really doesn't exist.
So I agree, one should trust their ears alone, but make sure that's what you're really trusting.
Pioneer VSX-1018AH-K, PDP-5020FD, DV-79AVi
Axiom M22s, VP150, QS8s
Sony PS3, surround backs