I had a similar issue with my old ep500 and ep600, I had to put the gain on the back of the sub almost as low as it would go, actually about 5 o'clock if I recall, and even at that the sub channel would end up around -3dB on my old Denon. As soon as I went to 6-7 o'clock the Denon would calibrate to like -10dB.
On my ep600, there was actually a problem and Axiom had to send me a new Amp, as it would jump like 20dB's just with the smallest adjustment, 5 o'clock to between 5-6 o'clock. I would just run the built in pink noise and try to auto adjust using my Radio shack spl meter, and it would jump 20dB's...crazy.
dj, I would start almost to the smallest gain on the back of the sub, with flat setting, and bypass on crossover, if yours has that knob.
Also, keep in mind, after the initial sweep from the primary seat, you can stop the calibration and save changes, go see what the levels are set to. Once you find the right adjustments, do it again and continue on with the other mic locations, you don't have to go through all 8 locations each time.
M80s-VP180-QS8s-EP600-2xEP350 Denon3808 Outlaw7700