I am a long time HTPC user. I ran XBMC for over two years. I however only used XBMC for media management; I used an external player for playback because XBMC did not produce the video and audio quality I wanted. I stopped using XBMC about a year ago and switched to JRiver MC17, it has a superior playback engine and I can use the external codecs and renderer I like.
I rip everything to mkv, I have upwards of 400 movies and many TV shows. IMO, mkv is the best way to go.
XBMC is not capable of bitstreaming audio so it just outputs PCM which your receiver sees at Multi-Channel. The problem with this is that the audio gets processed by the Windows mixer which unfortunately is sub optimal. Since you are only getting two channels, either Windows is set to two channels, or XBMC is. You say you checked Windows so you need to confirm the XBMC settings. Also, make sure XBMC is using the correct audio device for playback; you probably have more than one so ensure to set XBMC to use the device you set to 5.1 in Windows. I am not familiar with Onkyo receivers but it too may have a selection (my Denon does) to choose the number of channels.
I use Anydvd HD, to decrypt. I use Clown_DB to extract the main movie (no menus and extras crap) into it's basic audio and video streams. I then use the MKVtoolnix package (MKXmerge) for creating my custom mkvs. This system is great because I can pick and choose precisely what audio stream, language, sub titles I want and nothing else. Another great little tool is MKVchapterizer which adds chapters to any mkv, it's great for shows that have none. Finally, Mediainfo is your best friend, it can open just about any media file and show you all the details of the audio, video subs etc... (make sure you don't just get the .exe installer on the main page, it has OpenCandy adware, get it from the downloads page). Finally, if you don't want to use all the utilities like I do above, try MakeMKV. It's an all in one package. I however prefer the control I have by doing it myself.
http://www.slysoft.com/en/anydvdhd.htmlhttp://www.clownbd.com/http://www.bunkus.org/videotools/mkvtoolnix/http://code.google.com/p/mkv-chapterizer/downloads/listhttp://mediainfo.sourceforge.net/en/Download/Windows Get GUI version without installer
http://www.makemkv.com/On another note, there has long been special builds of XBMC that supported proper sound playback in windows using direcshow so you could get real DTS-MA/TrueHD (bitsteamed) playback and even 24bit audio. Back in version 10, it was called DSplayer. Now in v11 there's an AE version that a few developers are working on.
For me picture and sound quality are paramount, so I use external codecs and renderer with MC17. Even though I always did bitstreaming in the past, I switched to PCM Multi Channel soon after I switched to MC17. MC17 supports WASAPI Even Style which complexly bypassed the Windows mixer. Doing so allows me to enable the VideoClock feature which eliminates dropped frames when movies are not at 23.976 fps. It's great when watching European shows. Also, I get full 24 bit and up to 192KHz music playback with my flac music files.
Hope this helps.
