Plinko,
There is no intrinsic flaw of double-blind or A/B/X testing. The point of the latter is to simply consistently hear a difference, any difference. If you correctly pick A over B seventeen times out of 20, rather than 10 out of 20, which chance or flipping a coin would predict, then you can conclude that A sounds different than B.
Of course, anyone can purchase whatever they want, and if they know they are listening to a $1500 CD player, they have a psychological expectation for it to sound better. But the point of double-blind tests is to remove the effects of variables like that, so you conceal the two CD players under comparison, equalize the output levels, and call one A, the other B. If you can detect differences reliably, beyond chance, between the two, then a difference exists and you can explore that further with more tests. In my tests, once I equalized the output levels and ran two identical discs in sync, switching instantaneously between each, I could not reliably distinguish one over the other, or identify A over B beyond what flipping a coin would yield. Nor could any of my colleagues.
This type of testing forms the basis of all medical studies of drugs, to isolate the real effects of a drug from the patient's and the experimenter's expectations, which is why they give a sugar pill to a control group and the real drug to the experimental group. No-one knows who gets the real drug and who gets the placebo (the sugar pill).
By the way, my background is in journalism and I've been writing about and participating in double-blind and A/B/X testing for 20 years, long before I joined Axiom, when I was an editor of audio-video magazines. Axiom gives me enormous freedom to write about these subjects, far more in fact than I had when I was a magazine editor, where the publishers and advertisers often impose considerable pressure.
Regards,
