Martin, welcome to you signing in from the far north(I had no idea what Nunavut was until I checked the Google maps). The difficulty in analyzing your question arises because we don't know what you consider to be "decent bass". The basic concept that applies to setting the sub level is that it shouldn't be so loud that it draws attention to itself; the impression should be that the sub isn't working, but that the speakers are putting out more bass. My setup is calibrated in this way, but you may be dissatisfied that you can't "hear" that sub that you paid so much for. You've already bumped the sub level up more than the Denon auto-calibration set, so it would seem to be sufficient in general. You haven't described the location of the EP500, and that can also affect the bass level significantly.
As to the crossover, try setting the M80s to "Small" and 80Hz regardless of what the 3808 set them at. The bass assignment should be LFE, so that the EP500 is responsible alone for any LFE and redirected bass below 80Hz or whatever other crossover frequencies were set.
