The main goal of calibration is to set the speaker levels relative to each other. I would suspect that an auto calibration routine would also adjust to a set loudness (in db) level, so, in theory, both sets of speakers should be at the same loudness level. Too keep things fair, you should verify this with an spl meter at a fixed position (tripod, not hand held) where you will be listening. The reason for a tripod is that slight variations in the position of the spl meter will give different loudness readings.
As I understand it, equal volume/loudness between the speakers being evaluated is critical to the fairness of the test.
Speaker positioning is also critical as it effects how the speakers interact with the room and thus what you hear at the listening position. Harmon went so far as to design a fast rotating platform so that the speakers being evaluated were at exactly the same position.
The only way around the latter issue is to do a comparison, swap the speaker sets around in position, and repeat the comparison.
Taking notes would be important as well.
