Hi Mind. . .
XLR connectors were developed for professional broadcasting to enable running hundreds of feet of cable without picking up any hum or interference. For domestic applications like yours, there is no difference in audio quality between the XLR and the RCA connections.
Note that adaptors of all types can be "fiddly"; i.e. sometimes they don't seat properly and might be intermittent, the same way headphone jacks may get intermittent. It's just a spring-loaded connector inside the 1/4-inch phone to RCA adaptor, so make sure the adaptor is fully inserted and seated.
In that respect, XLR connectors offer a more solid, less "fiddly" connection, but audio quality remains the same.
Regards,
Alan
