For the connection from my DVD to the receiver, do I have to use the optical/coaxial output to enable dolby digital or dts audio, or is it possible to get 5.1(or better) audio with just the L/R stereo outputs?

yep, you need that optical or coaxial cable, whichever one you prefer. You should not notice a difference between the two, so whichever works better for you.

RCA style L/R connections will just give you those 2 independent channels: left and right.