The Dolby modes are a compression scheme (sort of like MP3) to allow the space for the 6 channels (5.1) of audio to fit on a DVD. With SACD or DVD-A the audio isn't competing with the disc space for video; compression is not used and so there is no need to set your receiver to these modes.
Each of the six audio channels is outputed from your players analog 5.1 outputs as un-compressed PCM audio. So you can set your receiver to 5.1 analog or bypass mode to hear the audio as it was recorded, without further processing or changes necessary.
As Mike pointed out, with this setup your receiver typically does no bass management so you will need to setup your speaker sizes (Large vs. Small) and crossover for the sub in the player.
Just make sure to set your player for multichannel out (SACDs can also output as a standard 2 channel CD), and that your receiver is using the multichannel input as well. If you aren't sure the sound is coming through the 5.1 cables vs. the optical, you can temporarily disconnect the optical to ensure you are using the signal path you want.
