if i was playing something at -40 to -23 or so dBSPL, i too would find the level so low that i would have some difficulty hearing clearly; unless it's the Energy's that have a problem.
the levels i was talking about for movies is the same level at which they play movies in the cinema.

i had Energy 22 Ref. for about 25 years, and with some program material, they would distort heavily for some seconds, taking in some 500 Watts each. the rest of the time, they were great,only lacking in deep bass, below about 32 Hz.

in room fr. resp. was about +2 to -3 dB 22Hz-18kHz.

if your yamaha is rated for operating at 4 Ohms, then it should work ok, but you may not have enough power.
why don't you start with the Yamaha and see?
it you hear any distortion while playing music of movies, then you should go for high power amps; not forgetting that doubling the Watts gives you only 3 dB more SPL.
i can't see how you would not be satisfied using the XPA-3 amps.