Axiom Home Page Forums General Discussion The Water Cooler Who reviews the reviewers in the Audio Industry?

Here is an example of why I personally have a problem with the reviewing industry - it's dismissal of blind listening tests. Their reasoning is the A/B test does not allow enough time to notice subtle differences.

Fine - then set it up where you have a 1 month, 3 month or 6 month blind test. All one needs to do is have the will to do it. But, again, it doesn't fit the narrative.

Rich - I have some funny stories about financial industry guys. Maybe sometime over drinks.

Like Rich, I am always skeptical - about everything!

I always read the written word and compare with vendor and reviewer measurements. Many times that puts the reviewer's description into context. Other times I get confused because the two don't mesh.

Bottom line is I always need data I can trust or I just procrastinate and it doesn't bother me. I learned how to do that in a Decision Science course. I also learned you can't judge a decision by the outcome but rather on the data you had when you made the decision and the process, tools and people you leveraged.

So even though I am a teen living in an adult body, I am wise beyond my teen years. And humble also.

procrastination ... I've perfected that one ...

Congrats. You are like Merkel, Eisenhower and Franklin.

This is an interesting topic, thanks to Craig for bringing it up. Back in the beginning days of Axiom, Sound Canada (which later became Sound&Vision Canada) did their speaker testing at the NRC and it included a full set of anechoic measurements and double-blind listen testing. As it turned out this was the peak for reviews based in science.

The reality is that speaker testing involves a lot of expensive gear and a lot of time. To do a full set of amplitude response curves requires an anechoic chamber and enough sweeps to get a total radiated sound power curve, that is expensive and time consuming and you are really just getting started. To do proper double-blind listen tests you will either need a shuffler, like Harmon has, or you need to repeat all the tests with each product in each position to weed-out the position error from the results. Then there is the testing to find out where the upper limits of the output exist and what happens when you reach them. Since this sort of testing involves destroying a few speakers along the way it is quite impractical to do as part of a review. Given all this it is not surprising that most “professional reviews” are really just “professional opinions”, and in many cases just “opinions” .

Of course the double blind, ABX test is the purest form of a test. But - let's explore options that would make current testing much better. We should have a standard as follows:

1. The reviewer has a reference system. This reference system doesn't need to include $500,000 worth of gear, but it should be high quality.

2. All components should be hidden behind a speaker cloth style barrier to prevent being able to see the component.

3. When a new component is introduced into the system, someone else sets it up. The reviewer has NO idea which component was swapped out, and what was swapped in. All he/she knows is a change was made.

Listening is done, and results posted, before the person is allowed to know the component. This method is very easy with speakers and amps.

Pre-amps and sources are more challenging, but still possible.

The issue is - none of the magazines is even willing to TRY. They tell us that one HAS to have the visual cues in order to do a proper test.

As Ian said - now it's just an opinion, and it's based on price, company name, personal bias and MAYBE some sound quality.

. .. "As Ian said - now it's just an opinion, and it's based on price, company name, personal bias and MAYBE some sound quality." . . .

AMEN !!!

How do you judge soundstage amongst multiple sets of speakers?

I've dropped non-Axiom speakers right where my M100s are located. Imaging/soundstage was terrible. I moved them further back and closer together and although they were no M100s, the imaging vastly improved. So did bass.

How do you reconcile that in double-blind testing? You'd need to know where to place the equipment under test before-hand I suppose.

Guys ... let's forget the ABX model, as it is not a realistic way to test a speaker or other component in one's home, which is where reviewers do the test.

Let's say we want a blind test on speakers. All that needs to be done is to have someone spend some time to get the best location for speakers in the listening room.

This isn't that hard to do. We audiophiles do it every time we buy speakers.

Then, without the reviewer having any contact with the person who set it up, he spends as much time as he wants listening to the speakers.

Let's call them ACME speakers during the review.

He does a complete write up, having NO idea what the speakers are. The review is published.

The next month issue will then have the reveal.

This would allow the reviewer all the time he wants, and an honest, unbiased review posted.

Everyone wants to complicate this, and it's really simple.

The audio magazine industry has been weeded out by natural selection ... the survivors are the ones who support their advertisers ...

This will slowly change as the cost of getting information out to the public has come down significantly. There are a lot of audio people who value honesty and reputation over money.