Toby,
Those are room modes -- plus/minus 10dB swings are nothing rare, and unfortunately you have them. What are the approximate peak and valley frequencies, as judged with AVIA's low-frequency sweeps? Anyway, in order to simplify the trouble-shooting, I would do the following:
1) Set all speakers as SMALL; bypass the crossover on the VTF-2.
2) Set the subwoofer crossover on the receiver at 80Hz.
3) Try several feasible locations for the sub, testing it with the AVIA sweeps after basic calibration.
4) In each location, try both "max extension" and "max output" modes on the VTF-2 (one vs. two ports open) -- the sub has to be recalibrated after changing the setting.
5) In each location, try both positions of the sub's phase switch.
6) In each location, try several different sub crossover frequencies on the receiver.
I also second the suggestion that you FAX a diagram/measurements of your room to Dr. Hsu (see their web site).
Good luck!
