At least with my AppleTV, even though it is connected via optical digital only (TOSLink, just like yours), you can control the volume from it alone (like from the other room, via AppleRemote). In other words, unlike a DVD player, Apple does tinker with the "levels" coming out of the digital connection. That might have something to do with it.
I've never had any problems like you're describing, but my receiver is a Pioneer, so that's obviously going to be a bit different than your Yamaha.
But one thing I'd try first is to replace that optical cable. I've had bizarre problems with optical cables in the past that were always solved with simply getting a new cable. All it takes is one too-aggressive of a tug or a too-sharp of a bend and you start to break fibers. Break enough and weird things can start to happen.
Something to try, anyway.
