Peter (pmbuko) did a diagram of how to set this up safely about a year or two ago. Basically, you don't want to use a speaker switcher, because you've got no guarantee the receivers won't overlap for even a small amount of time. Overlapping receivers would be very bad, from what I've heard.

I am the Doctor, and THIS... is my SPOON!