I detest being put on speaker phone. I like it even less when the person calling does not inform me I'm on speaker. Whenever I am called, and I think I'm on speaker, I always ask the caller to pick up the phone. Conversely, when I call someone and I have them on speaker (in the event I need my hands free or someone else is in the room) I always, always start the conversation off with "hey I have you on speaker, and I have #### here with me". It's just common courtesy as far as I'm concerned.
As far as your personal information being discussed without your consent, I agree with you. You personal business is yours, no one else's. If for some reason you feel the need to share that information, then that's your decision. For a government employee to run off at the mouth without your approval, I find that a gross error in judgment. But, and I emphasis the 'but', it is your responsibility to inform the caller that there is an audience and to be discrete with your personal information. That is unless this caller knew that your care giver was there and you had the caller on speaker.
