I didn't think the PS3 downmixed and matrixed the multi-channel PCM when being output over the optical link. But if you're hearing clear dialog (the center channel), then it must at least downmix.
You're right, a new receiver would be the "best" solution, but until I got mine I was very happy with the optical outputting Dolby Digital or DTS. Honestly, I was a little disappointed with how little difference I heard after the upgrade. But getting the two extra surround channels with 7.1 which prompted me to buy the QS8s, was an ear opening experience.
While technically, there may be enough bandwidth on a S/PDIF link for Dolby TrueHD or DTS-HD Master Audio, that is not what you get. When picking one of the lossless formats, the PS3 selects the "core" lossy portion of that track and sends that to the optical port.
The way you should have your PS3 configured is for optical output with bitstream. Then pick the highest quality audio track on the disc which is not PCM. If a Blu-ray has both Dolby Digital 5.1 and DTS-HD Master Audio, pick the later. The Core of MA is a 1.5 Mbps DTS track, which will top Dolby's lossy offering.
