Just to start with the basics, are your speaker levels calibrated either by the receiver's auto-setup or with an SPL meter? It's less common than before, but some people still calibrate by ear rather than with those tools. If you've calibrated, then that eliminates one possibility. Also, where is the crossover set in the receiver for the center channel? Hopefully it's set to small, not running full-range.

So you only notice voices distorting, not other sounds?

Also, can you describe the distortion? Someone else with more knowledge, like JohnK, will have a more complete response than me, but that should give them some more information to go off of.