>>Could the setup be wrong?

The EP175 is a fairly small sub to use with M80s. If you can adjust crossover frequency for the mains separately (eg. HK receiver) you could try taking the crossover down to 50-60 Hz so the M80s do a bit more of the work and the EP175 just has to handle the lowest notes.

Assume it sounds "not clean" only on the loudest parts of the soundtrack ?

M60ti, VP180, QS8, M2ti, EP500, PC-Plus 20-39
M5HP, M40ti, Sierra-1
LFR1100 active, ADA1500-4 and -8