How is it double blind if one person is clearly able to tell a difference when he sets the speaker?
Double blind studies are used to rule out actual differences from prejudice.
If you don't know the how's and why's, here's the short of it.
The "double" means both the tester and testee, are unaware of which sample is being tested. Or if the tester does know there's no way for that information to be communicated to the testee (that means no forms of communication between the two, not even non-verbal forms like body language).
The two samples are presented. The testee is then tasked with identifying each one uniquely after each switch. If he is only correct around 50% of the time, then it is just chance. But if one sample can be identified a majority of the time, then there is a difference.
Two very difference speakers will obviously be able to be identified uniquely.