Hi 1sweetspot,
Don't trust Audyssey. With a problem such as this, I would do a full reset on the Denon (read owners manual) with so many factors possibly in play. This will change everything back to the factory/default settings. This would also disable any whacky dynamic range settings which could also compromise output (if enabled).
Once you do the reset, go back in the Denon settings and re-enter the trim values, speakers set to small, 80hz crossover you already have manually and skip Audyssey for now. Re-run your tests. The best way to verify these settings is with an SPL meter as Alan suggested. One of the best tools an audiophile can make. And yes you can try increasing all the values and see where this takes you if you plan on doing everything by ear.
As for your subwoofer, the pre-out signal it receives might be low with a -11.5 setting. Turn down the gain on the subwoofer control and increase the pre-out setting of -11.5.
