Okay, if at that level there was massive distortion with the 805(regardless of whether it actually fell into the "clipping" category)then your sample of the 805 is pretty clearly defective and should be returned for a replacement unit. Just to check the M80s further, you might connect the 777 to them to verify that they're working well, as is almost certainly the case. The 777 should have no problem with them, at least at moderately high levels(it's THX Select certified), as should also be the case with an 805 in good condition.
As to your previous 805 vs 777 comment, yes there'd be no reason to expect that the 805 would sound better if just the basic amplification was being used, without the different processing in effect. The just slightly higher power capability wouldn't become an audible factor in most instances.
