I can think of two reasons why you should not: 1. you may accidentally turn on both sources and that could be trouble for one or both sources, 2. the output of one source could short and thereby short the output of the other source.

A switch is a pain and may affect the sound. A good option would be to use banana plugs at the speaker terminals and simply disconnect one or both plugs connected to the source that is not in use.

House of the Rising Sone
Out in the mid or far field
Dedicated mid-woofers are over-rated