Sounds like you want a DAC. I have an iBasso D10 and love it. Check the ibasso website, the D2 and D4 are well received. I like my D10 because I can use it on my Macbook via USB and optical out, and yes, optical sounds much better w/ the right recordings. It's dead silent, with no music turn it up loud and you hear nothing, no hiss or noise.
Also try used. I bought a Leckerton UHA3 from a local kid via the head-fi.org classifieds. Cost me $125 IIRC. That was a great, small starter amp/DAC.