Axiom Home Page Forums General Discussion Technical Questions Critical Listening

I'm still not 100% sure where I stand regarding listening tests. Audiojunkies.com had a link to an editorial by Robert Harley of The Absolute Sound:

Blind Listening Tests Are Flawed: An Editorial

And a more in-depth essay by him on the subject:

The Role of Critical Listening In Evaluating Audio Equipment Quality

I'm only part-way through the essay, but it seemed worth posting. If nothing else, we can collect more points and counterpoints on the topic in one thread.

PLEASE, someone correct me if I'm wrong, but his general theory is "how can you expect the results to match your hypothesis unless you give the listener the hypothesis in the first place?"

If that is indeed what he's saying and I didn't just misread or misunderstand his comments - that's so stupid it makes my brain ache.

"But Chief Clancy... how are we supposed to get the witness to identify our accused unless we tell him #4 in the lineup is our suspect?"

That's it... I quit... last one out of civilization, turn out the lights.

Bren R.

Sure Charles, more of the same sort of drivel that Bob Harley has been spouting for about 20 years in his "Complete Guide to High-End Audio" and various illusion-oriented audio mags. If the tests don't support what's "obviously" clear to them, then it's the tests that have to be at fault, not them. It doesn't work; facts don't cease to exist when they're ignored.

I guess my hangup is that knowing what the products are that you're listening to doesn't always lead to you liking the sound of the higher-end component. I'm wondering how the mind decides which one it prefers in those cases. Is it simply the more inconsistent methodology making the results more erratic?

The problem is that when you know the product(s) being used and if you already have a preconcieved notion of one of the said products, then you will be relating/comparing all tests back to that one (or more) item(s) you already know, creating a biased test either in favor or not in favor of the known product.

So let's say someone is told that the components have been switched but nothing was in fact switched, and they decide they prefer one over the other, even though it's the same thing. Is this trick okay, or is it accepted that the listeners have to know when different components are in use? I've never read too far into how listening tests are done.

In a double-blind test, ideally the listener will never know whether the same or a different piece of equipment is being used to reproduce the sound. If you're doing a psychological experiment, you can, of course, tell them you're switching components and not switch them. In this way, you could gather data about how much that affects their actual perception.

Echoing Bren's comments... it seems that Harley's conclusion is that, 'if the test is to determine whether an error is audible, and given that the error exists, if no error is detected, then the test mut be faulty.'

It seems that this totally misses the point. If the purpose is to detect the anomaly, then perhaps double blind testing isn't the way to go. But if the purpose is to determine if the anomoly is detectable, thendouble blind testing may be an apporpriate means of doing this.

The purpose, it seemed to me, of the test was to determine if the compressed codec proposed for use would be detectably different (by the audiophile, let alone the commoon man) from the original, uncompressed sound. The test determined that the error was indetectable - i.e., good enough for broadcast standards.

If the test was aimed at challenging audiophiles to find and locate an error, then perhaps it might make sense to tell them which sound was compromised and permit them an opportunity to first listen to the unadulterated signal, then listen to the error laden one and try to tell where the differences lie.

The first test is a practical one, for use in the real world, whereas the second test seems academic, at best.

I think that Harley has really drank the coolaid on this one. His example to "prove" the case - that monoblocks, tube amps and a $99 receiver all sound the same in a double blind test - does nothing more than to reveal HIS obvious bias.

I like how he concedes that the double-blind test has been proven useful in medical testing but it must be suspect because it has amassed a pile of data that contradicts what he believes to be true.

It's a classic battle empiricism (the idea that we can observe the world and evaluate those observations in relation to facts) and post-modernism (which holds that facts are fluid and elusive, so that we should focus only on our observational claims).

Here's the thing.

You can criticize anything you want about any particular double-blind test: The amount of time a person is listening, whether the audio material is familiar, whatever.

Fine. Make a double-blind test that addresses those problems.

Harley can write for pages and pages (and pages and pages), and not a single thing he writes (that I've come across) addresses the primary point of double-blind tests: you know, the double-blind part.

At the extreme, his arguments against "objectivism" could be taken to mean that no one should listen to his opinions, either. If there is no objective viewpoint, then there's simply no point accepting his viewpoint, either. It's a common problem of post-modern flavored thinking.

I beleive that same line of thinking is why 'yall ended up in Iraq.

I say, well done, gentlemen.

Carry on.

A noob question I can't get out of my mind.. as I peruse the various forums/retailer's/opinion's all over the net.

Are there not testing equipment that could measure the sound produced in various tests that would show a measurable difference, even if that difference isn't noticeable to a listener? e.g. speaker wire.. If all you change is the wire, and you play the same set of tones and you can't measure the difference.. it would seem to me you wouldn't hear a difference either

Maybe I'm too used to the PC world were comparison's are less subjective.. or maybe such exacting measurements are only possible with very expensive testing equipment.. I don't know.. but how much better can a 40 000 CD player be compared to a 1000 CD player??

If you can't measure a difference, there's no difference. At least, not in what you're measuring.

We have a pretty good idea what's important in sound, but perception is a complicated thing. Of course, if you measure the sound wave, and it's absolutley identical in two instances, it's pretty insane to think there will be an audible difference.

Things can get complicated, though - what kind of trade-off should one make between off-axis response in speakers vs. overall "flatness" of response? But with wires, etc., it's generally pretty simple.

But let's assume there's some bizarre thing we're overlooking about the signal we're measuring. Or magic. Or whatever. The beauty of double-blind listening tests is you can still test for differences due to unknown factors. It's all in the listening!

And a $40,000 CD player is simple insanity, unless it also functions as Great Art or something.

I have to admit I stopped reading his essay a few pages into it. If the following is already covered by him, I apologize.

Double-blind listening tests may be our best shot at higher audio performance, but I'm not sure we've done enough kinds of research to work out the weaknesses. For one, they don't recreate a normal listening experience. No one listening to their own system is unaware of what hardware they're running. They're also not instantaneously switching between two components for comparison. As for me, I know I end up noticing new details quite often when I'm engaged in the content, not necessarily listening for sound quality differences. What if our best listening isn't when we think we're listening critically? Then again, how much research can be done on that? It's just that so much of our ordinary listening has us preoccupied with the lyrics, or with the story. Maybe the general impressions we construct over time with this casual kind of listening have more value than critical back-and-forth double-blind.

How many audio companies DON'T utilize double-blind listening tests when designing their products versus those that do? I'm sure the ones that were created out of magic have higher price tags, but I wonder if it's guaranteed that they'll perform under the level of those that have benefited from "proper" research.

Most of you are much more studied than I am on the topic, so I'd love to hear of the research that's been done.

This topic reminded of this article I read a few months ago.. not sure this is the exact article, but it's discusses the same research study..
Wine Test: $5 bottle gets Tastier when it's $45

This study showed an actually difference in brain pattern's when drinking what was believed to be $5 and $45 a bottle wine.. even though it was really the same wine.

This could be taken to support the idea that people hear a difference, because they want to hear a difference, not because one exists.

Amen to that.

For me, my best listening is when I could care less whether I may be listening to compressed source material and enjoy the music for how it affects me and lightens my mood.

You guys are right. Now that I think about it, I do my best listening in the car. I'm much more concerned about simply enjoying the music while I wait in traffic. Plus, I can sing along as loud as I want and use whatever surface as a drum.

There's absolutely no reason for them not to. The only necessary difference - you don't know what hardware you're listening to. But why should that matter?


What do you mean by "best"? If you mean "detecting small differences", then you want typical DBT conditions. Highly focused attention, immediate switches between stimuli, etc. These are all designed to MAXIMIZE our ability to detect small differences. A century of auditory research has converged on these procedures. The only reason people say these rules don't apply in AV World is because the results go against their preconceptions.

If by "best" you mean most enjoyable, the kind of listening we should be doing for ourselves, listening to music instead of listening for differences in sound - then no, DBT isn't the way to go. But that's a completely different issue.

As for noticing subtle details...If you're talking about discovering them in music, you'll want normal listening conditions. If you want to detect differences in how that already-known detail is reproduced with Speaker wire A vs B, you want DBT.

A century of research and not one specific study cited? That's fine, but I was kind of hoping the idea that direct comparison could play with your perception of sound quality would be taken out with an elephant gun, not a stealth bomber sans payload.

Oh, and listen for the new Negative Orange single, "Stealth Bomber Sans Payload" in August with the full album following in the full.

But when's the Negative Orange tour?

Detecting minimal differences is a common goal of all sorts of perceptual experiments...and that was what I was referring to. That's a very different issue from "perception of sound quality" - or could be. The first is well defined, the second could mean quite a few things.

I'm literally talking about tens of thousands of studies. I wasn't making a specific point about a specific finding, I'm just talking about the general state of the procedures of an entire field of research. I mean I could have given:

http://www.perceptionweb.com/

or a dozen other journals and said "start at volume 1", but that's not very helpful.

More to the point of the thread, if Harley thinks one or more of the standardized procedures is faulty, that's fine. Design a blind test that addresses that. But keep it blind. It's simply silly to suggest you need to know what equipment you're listening to.

By no means am I suggesting that ALL equipment reviews MUST included stringent DBT procedures. I'm more than happy reading the opinions of experienced listeners with a good track record, especially for things that are likely to have large, noticeable differences (e.g., different loudspeakers).

But if someone tells me that cables are "danceable", but blind testing repeatedly shows no differences in cables meeting certain minimum specs, I'll trust the blind test, thank you very much.

Charles, I'll cite a couple of the things that I have cited in the past here and might be informative and amusing to you. The Stereo Review amplifier tests which enraged some audiophiles, but opened the eyes(and ears)of others, still are unchallenged(i.e. unchallenged by contrary results from later tests, not simply by a stubborn refusal to accept facts)and illustrate the contrast between some rather flowery language used to describe sound differences in open listening before the tests and the actual blind test results. Among other notable results was that the $220 Pioneer receiver was indistinguishable from the $12,000 pair of tube amplifiers.

Our Alan has been involved in many blind test sessions and you might find interesting this report of the testing that defeated Ivor Tiefenbrun of Linn where Alan was present(still wonder if he ever got his watch back).

On whether the listeners know if a change actually has been made, this isn't necessary, but the ABX test method requires an actual change from A to B before trying to determine which of them X is. A different blind procedure, same/different, does sometimes have the same unit played for both samples, and a fair amount of the time the listeners incorrectly mark the same as being different.

One amusing incident that speaker designer John Dunlavy has occasionally recounted on the internet occurred when in a supposedly non-blind comparison between regular lamp cord and a very expensive boutique speaker wire, the "technician" behind the speakers apparently busily making switches actually left the lampcord connected at all times. One of the several "experts" present at Dunlavy's test, after the expensive cable was supposedly switched in, commented to him: "Omigod John, tell me that even you can hear that difference".

In one of Dr. Toole's excellent papers which touches on the necessity of blind listening tests(at one AES meeting where he spoke on the subject, he stated, using a bit of hyperbole, that "If you can see what you're listening to, you can't hear it!")he shows at p.10 that even when items where differences actually exist, i.e. speakers, are involved, "huge" differences in open listening are greatly narrowed when the labels and price tags disappear and the listening becomes blind.

Definitely. I'm really only throwing stuff out there for the sake of discussion. Magic costs a lot more. I certainly prefer being able to afford the good stuff.

Tours of the Negative Orange Musicadium are offered year-round.

As always John, you come through with the goods. I'm part way through Dr. Toole's article. Its quite thought provoking.

Oh, I missed JohnK's response. Thanks for making me come back to read it.

You are welcome. Do take the time to read the Toole article. There is lots of interesting stuff in there.

One of the most interesting to me is his discussion of the ability of different individuals to select good over bad speakers. He says outright that people have to be trained on what to listen for. To me, that just screams introduced bias. I'm still pondering that and will read over the section again.

The other one that surprised me was on resonance. It seems that low frequency resonance is detectable at much lower levels than high frequency resonance. This is the exact oposite of distortion.