If one driver isn't connected right now; it'll move in the opposite direction. That is when the powered one pushes out, the volume of the enclosure increases so the unpowered one gets sucked in. But if you were seeing this while both drivers were supposed to be working, then yeah, I think one's wired out of phase...but which one?

Pioneer PDP-5020FD, Marantz SR6011
Axiom M5HP, VP160HP, QS8
Sony PS4, surround backs
-Chris