I have one port that is eternally loose on my switch. If you can see inside to see if where it's safe, drill and tap a small hole near the port. I then tightened a wire-tie (zip cord) around the hdmi cord and put a screw through the length of it into the switch to secure it into place.

A little cleaner looking than wrapping the wire tie right around the box but that is another option.

With great power comes Awesome irresponsibility.