You're using the TV speakers as "center", right ? If so, I guess the question is whether you have them turned up to the same sound level that you would use with a dedicated center channel speaker.
The center is primarily intended to "anchor" dialog on the screen when you sit off center, so I imagine that as you reduced the sound level of the center (like I get tempted to turn down my crappy surrounds
) you would get progressively less anchoring and would hear more of the sound from whatever side you happened to be sitting on.
Are your TV speakers getting a center channel feed or just a blend of all channels ? Either way, I guess turning up the TV speaker sound level would be the first step.
Good luck...