The M80s will play louder than the M5s. But you may find positioning them in the room is a challenge and you may not be able to get them to acoustically disappear.
With that amp and subs, I can't see a benefit with M60s. M60s may be a detriment actually for the same reasons as M80s.
The M5s would sound terrific with as many subs as you want. I've used them with up to 7 subs. Fantastic!
The M5s are far less prone to position error than the larger Axioms. They are as refined as the M100 and active LFR and far easier to succeed with quickly.
Open floor plans should be treated as a single volume, but since you're sitting closer to the loudspeakers within that volume, you don't necessarily have to upsize all the audio components. But if you like it loud, err on the side of more power and bigger speakers.
As my hearing is almost as good as it ever was, I would approximate that at ~ 6 feet no more, if not less than 45db SPL @1kHz, 53db @ ~ 8kHz, 65db @ 100Hz, and so on, below which my subs would take over.
https://en.wikipedia.org/wiki/Equal-loudness_contour That speakers can achieve absence of their audible presence is certainly a good thing, especially if this happens along with a spacious soundstage and other characteristics that create and transport the listener to realistic sonic realm. That's why I've always found Enjoy the Music's performance check list extremely useful for evaluating hardware. See the bottom of this page.
https://www.enjoythemusic.com/magazine/equipment/0214/axiom_audio_m100.htm
