Where did you set the crossover? At the sub?
In the receiver.
I did some more testing with speakers set to small and the 20, 40, 60, 80hz XO's, and I get the best, smoothest response with small speaker setting and 60hz crossover sub in a 15 hz tune, which is where I thought it sounded the best without knowing any of this graphing. It isn't much different than what is above. It just goes to show you can trust your ears, to a certain extent.
Now to play around with the RTA software to see if it comes up with different readings and then more full spectrum graphing with 5.1
