Bringing back the golf theme. (What is it with golfers???)

Two men are enjoying a game of golf, but keep finding themselves held up by a slow pair of women ahead of them. After a while, one of the men decides to go up and ask the women for permission to play through. He makes it halfway to where the women are standing, then turns around and hurries back to his friend.

"I can't go up there. One of the women is my wife, and the other is my mistress! You'll have to go ask them."

So the other guy starts to approach the women, but quickly comes back and says, "Small world."
I can explain it to you but I can't understand it for you.