You could always go with a speaker switch box. It may be the simplest way.