Depends on the AVR inputs. You will not be able to get full hd audio over optical or coax (with any equipment).
Using the PS3 you would need an AVR with audio over HDMI (not all HDMI includes audio) to get HD audio sent as lossless PCM (the PS3 does the hd decoding). You would not need an AVR capable of decoding the latest codecs. Other player/AVR combos would also be able to use multichannel analog (using 6 or more RCA cables), also with the player doing the decoding. There is unlikely to be any noticeable difference in sound quality between the player or AVR doing the decoding.
