I've just set up my Pioneer VSX49TXi receiver with M60's, VP150, and QS8's. The receiver manual says to connect both analog RCA's and digital coax from the DVD player to the receiver, so that I can still play DVD's that don't have digital surround encoding. When I connected both and played Star Wars Episode I, it only would play in Dolby ProLogic II. When I disconnected the RCA's for audio, then it played in Dolby 5.1.

Is there a way to connect both and prioritize which output is played?

MC1,
My Onkyo has setup features in the receiver options to select which input mode will be associated with a particular connection type. The Pioneer should not be much different.
For example, i have both rca and optical inputs plugged into the receiver. When I select "DVD" on the receiver then my optical input is used. When i select "CD" the rca inputs are taken as the signal source.
Note that i use my dvd player for both cds and dvds but there is no noticeable difference b/w the analog or digital inputs on the sound quality. The receiver and dvd DACs are of such high quality that the limiting component is the cd recording itself.

On my lowly 45TX, there is a SIGNAL SELECT button on the remote, where you can select either DIGITAL, ANALOG or AUTO input mode. The former two options forces the receiver to use the respective input, which should solve your problem (I don't know where exactly the equivalent botton is implemented on the 49TXi's universal remote -- consult the manual). The receiver remembers the setting for each input.

It is funny, though, since I usually leave it to AUTO (default setting for DVD input), which chooses digital over analog when both are available but goes with whatever is available.

In addition, no matter what you play in the DVD player (DVD, CD, etc), the decoded material should be sent out of the digital coax output. You should have no need for using the analogue RCA jacks.

With that said, it is odd that the manual tells you to hook up the analogue connectors as well. I suspect, however, it is a mistake, or some kind of fail-safe for lower-end systems.

The most common problem people have with DVD players is not configuriong the digital output of the player for bitstream (vs. PCM) The result is only sending PCM audio out of the player, instead of Dolby Digital or DTS (which sometimes has to be enabled separately).

You successfully got DD, so if you leave the analogue connections off, you should be just fine.