Have you ever experienced what you describe or are you just seeing it in your mind's eye?
As I explained in another post, when something happens off-screen, the visual is off-screen too. So there's no discord taking place in the brain. If something is happening on the left of the screen, the audio is typically imaged in the center.
I say "typically" because I find with the actives, the image could be just left or right of centre depending on how it was microphoned and mixed. If I play that same scene with my 160, that same sound collapses to the center.
Try it yourself and see if you experience a brain schism.
