My bad. I was looking at the diagrams from the book shown above and thought that was the problem. Dynamat is pretty amazing stuff. That is what is used to dramatically quiet down road/engine noise in cars. You would have to line the whole outside, which would be pricey, but I am sure that some other dense and similar material would be effective with some loose, regular insulation. Double up on the drywall, or enclose the area around the duct with one layer before putting up the drywall would help as well.
