The crossover knob on the back of the sub, if yours has one, needs to be turned to the highest setting, or "bypass", if that is an option. Since the receiver will be handling the bass mgmt, you want to get that crossover knob out of the way. Normally, with the EP500, you can start with the volume (gain) knob pretty low. I think I leave my EP600 volume knob about at 20%, and then after running Audyssey the sub dB level on the receiver will be closer to 0dB, with the other speakers.
Audyssey actualy is not what is setting your speakers to small or large, it measures the -3dB point of each speaker and reports that back to the receiver manufacturer, in your case Denon. If a speaker has a 40hz -3dB point, they will be set to large.
So it is not unusual for your m80s and vp180 to be large, mine was also. With that said, you still want to go into the receiver setup and set them to "small", and I would say a 60-80hz crossover.
Audyssey needs a totally "quiet" environment when run, with no noise at all. Also, placement is key for the various mic locations.
