In reply to:

Nevertheless, using the same CDs for a comparison where the only variable is the CDP seems to me to be a pretty good test.



No. The listener in your test still has an awareness of what the source is of the test. As such, psychological effects such as pre-disposition will play a HUGE part in the results.

A good example of this effect is speaker wire. Everyone that's purchased expensive speaker wire swears by their results. Sit them down in a double blind and you won't get anything close to a consistent conclusion.