Audessey and just about every other auto-equaliztion program found in the Denons, Pioneers and Yamahas etc. are notoriously bad at setting anything related to frequencies below 100hz. I would ignore any sub, delay or crossover setting and adjust manually. Simply not enough processing power in the receiver chips to find the optimal LF settings.
Here's a few suggestions.
1. For HT don't set the M80's to large because you are duplicating the LFE channel when using Dolby digital tracks as well as introducing phase and other artifacts when in stereo mode. I would suggest turning off the EP500 and just listening to the M80s full range for music if the Denon has such a setting.
2. Set all speakers to small and cross as follows (based on 1/2 octave above low -3db extension for the M80s and close to -3db the other speakers).
M80s 50hz
vp150s 80hz
qs8s 100hz
3, Listen for a few days and then cross all speakers at 80hz. The reason I am suggesting this is that the Denon power rating at 130wpc is likely not accurate for all channels driven so you may benefit sonically from easing the power demands on the amp by crossing the m80s higher. Also keep in mind that depending on the slope of the Denon, which is likely -12db/octave, you are still getting output above the crossover frequency redirected to the EP500 which brickwalls at 100hz. You can also try leaving the qs8s at 100hz.
Depending on your room and your listening habits I suspect you should find an improvement.
