i had this problem a few years back; the delay was setup right.

i solved it by hanging something (to diffuse reflections) a few inches off one of the walls at the position where this was happening.

after that, i never got that problem anymore.