My question is would I really hear much of a difference in sound quality with the M80's over my M60's?
No. you wouldn't hear much of a difference in casual listening with the possible exception of a bit more bass depending on how you run your present speaker setup.
The mindset is that the more expensive speaker, larger speaker must be an "upgrade" but this is a fallacy.
However based on your other comments it sounds like you may be sold on the idea to buy the most that Axiom offers so really there is no decision to be made as it sounds like you have already made it.
I am contemplating moving from the EP350 sub to the EP500 but for performance issues related to the EP350 that i've had for years.
