Optical and coaxial both carry a digital signal. They just use different cables. Since the signal is digital, you don't need to worry about signal loss affecting the quality on the other end (unless the cable is damaged). Therefore, I'm partial to coaxial, since aren't fragile like the optical cables.
Using the 5.1 outs on your DVD player might or might not be better than running the single digital connection to your receiver. It just depends which unit has the better decoder. You might want to experiment to see which you think sounds better.
