Leaving both switches down isn't an option, as it is electronic (you can switch from the remote) and hitting A turns B off first, then switches A on.

I'm guessing from the relays clicking when I switch that they are isolated, but I would research more before I actually do it.

I have asked Onkyo about it, and I haven't gotten a useful reply yet.