I can think of two reasons why you should not: 1. you may accidentally turn on both sources and that could be trouble for one or both sources, 2. the output of one source could short and thereby short the output of the other source.
A switch is a pain and may affect the sound. A good option would be to use banana plugs at the speaker terminals and simply disconnect one or both plugs connected to the source that is not in use.
Thanks MoJo for the reply. I do you use banana plugs and in the back of my my mind I was concerned about some sort of feedback finding its way back to the new Anthem receiver and messing it up if I had both avr's hooked to the same speaker. It is not that difficult to pull the banana plugs out so probably do that but being a bit of an old guy with an ancient back i will explore a some switches and see if there are any good ones out there that wont mess with the sound.
