While I agree that proper testing procedures are required for definitive proof of audible differences when dealing with the same class of speakers, there are situations where there is such an Extreme difference in quality as to safely make an immediate judgment of preference.
For instance, put a small cube speaker from a cheap HTiB setup up against an M80 grade floor stander??? An extreme example to be sure. True, in this case we are dealing with a modification to just one component without changing anything else like the driver, cabinet, etc. Here there is indeed argument for controlled testing.
However, just to be the devil's advocate..... What if Jake adds in these new crossovers and they change the sound in such a large degree that it is simply unmistakable. Unlikely but perhaps the new crossovers are tuned sooo way off for this particular speaker configuration that it turns them into the tinny sounding equivalent of a tiny cube speaker. Or, to be fair to Ninja, maybe it goes the other way around. Although an improvement to that scale is even more unlikely due to the laws of diminishing returns on a speaker of this caliber.
So no disagreement on the necessity of controlled testing for most situations but lets not say his experiment has no value just yet. I for one, look forward to his comments as he is the only one investing the time and money to do at least this much.
These are essentially my thoughts as well. I haven't heard this particular product, so I can't comment on what I expect the outcome to be, but to say there has to be a strict scientific method to even decide whether or not there is a difference probably over complicates the issue in the particular case.
If we're talking about the difference between the original components and premium level components, I can understand that need, as there is debate as to whether it makes any difference at all already, and the "voice" of the speaker is likely to be similar even if there is improvement. Same goes with comparing speaker wire or IC's or power cords or whatever. In this case, it's important to remember this is a bigger change than that. The topology of the network is actually altered here, with different crossover point altogether.
Like I said much earlier in this thread, I haven't heard an example of one of the designs that only changes out components, but I have heard the Onix Reference 1's both before and after, and the difference isn't subtle. (also a design that is all new vs. just upgraded parts) It was a gtg with 90 people in attendance and I didn't hear a single person claim they thought there was no difference. A few preferred the original, most preferred the upgrade, but nobody questioned that they were different. (and no, most weren't audio snobs that think $500 cables are better than $400)
I suppose you could give an extreme example where his room and front end were so awful he couldn't tell the difference between his m22's and a bose lifestyle system, but that doesn't appear to be the case here. It stands to reason that we can get a general idea from his observations.
Naturally opinions should be taken as such, and it's absolutely not scientific, but that doesn't make it un-useful. It certainly isn't a 100% decision as to which is better, but most of us are just curious to hear what he thinks of it. For that matter isn't that how most of us decided to buy our first Internet Direct speakers?