The vinyl itself is part of the issue -- overmodulated peaks on an LP don't distort as dramatically as overmodulated peaks on a CD -- so from the beginning there was more need to carefully control peak levels on CDs. This could be done in at least two ways -- lowering the overall recording level or compressing the dynamic range. I would have personally looked for some kind of soft limiter with behaviour more like that of a vinyl-based medium but that's just me.
The fact that the general public's listening habits changed (from serious listening to "listening while doing something else") is probably coincidental, but was another argument for limiting dynamic range and provided a very convenient solution for the challenges of recording high dynamic range music on a limited resolution digital medium.
It's interesting that nobody has focused on the changes in recreational self-medication over the last 30 years, from drugs which encouraged serious contemplation of music to drugs which generally made one unable to concentrate on anything
