If you have a sub, your speakers should be set to small. The term "small" has nothing to do with physical size of the speaker. It has to do with the crossover point for that speaker and how the "hand off" of the lowwww frequencies get passed to the sub.
Yes the M80's can handle low bass very well, I have them, but the sub has a much larger woooooofer and moves a lot more air.
Audyssey looks for the -3dB point of each speaker and reports that back to the receiver manufacturer during setup. Denon uses 40Hz, so since the 80's are "capable" and have a -3dB point of 40Hz, the Denon marks them as "large".
You still normally would set them to small. Now, I also bumped up the frequency which was set to 40Hz, to 60Hz, or possibly 80Hz. It is ok to bump up the crossover results, you just don't want to go lower than what Audyssey sets, as that gap won't be corrected.
Just because you set it at 60Hz, or even 80Hz, the low frequencies do not just get "cut off", there is a gradual slope/blend or "hand off" to the subwoofer. 60-80Hz is pretty low.
Yout want your sub handling frequencies down from that to say 20Hz or lower.
Answer to questions:
Your still getting killer bass when set to small.
If you set to small, LFE + Main would not be used, so no.
Turning the gain (volume) on the sub, or the level on the receiver makes really not much difference, it is the same speaker. I would start with a midpoint level on the sub, and after running Audyssey for the first sweep, if it sets your sub on the receiver to low (say -8dBs), bump up the sub and rerun the setup.
The first test from the primary seat when running Audyssey adjusts for distance, and works with the receiver to determine small or large. At that point you can stop the test and check your initial results, and make changes, you don't have to run all 8 positions first.
Leave the LPF alone at 120Hz, this is not the crossover and has nothing to do with your setting on the back of the sub. Audyssey recommends leaving that at 120Hz as there is "some" material during movies that goes above 80Hz and upto 120Hz.
By setting your subwoofer crossover knob to the highest setting of 150Hz, you are "getting it out of the way" or ignoring it, so the receiver handles your bass management.
Leave the sub crossover knob out of the equation, leave it at the highest setting.
For the 80's I would try bumping them upto 60Hz for starters and see what you think, and possibly 80Hz. For the vp150 leave it at 100Hz, you never want to "lower" a crossover point below what Audyssey reports, or you will lose the correction ability between that gap.
You won't get seat rattling from subwoofers unless your running them hotter than you should, get bass shakkers if you want your seats to rattle, or put a sub in your car.
