Can your amp handle all the M5 load?
Let's say you go M3 up top and M5 for surrounds. Imagine a chopper taking off on your left, flying above you and landing on the right. It'll sound like this:
THWACK!THWACK! THWACK!THWACK!
phhtt, phhtt, phhtt, phhtt.
THWACK!THWACK! THWACK!THWACK!
The mid-woofer and HP woofer on the M5 THWACKS! while the M3 just phhtts.
Yes, mounting OW directly against the wall and ceiling improves bass and lower mid-range performance. You could use bookshelves but having bookshelves so close to a boundary makes KABOOMs! Possibly your receiver can correct that.