Jasper, the volume control on the PM16 is a variable resistor which varies the amount of the incoming voltage(from the 6004 pre-outs in your case)which is let through to be amplified from almost 100% to nearly zero. There's nothing wrong with setting it all the way up, letting in almost all the voltage from the 6004, if adjusting the main speaker trim settings(also variable resistors)on the 6004 can achieve a balanced level between the center and mains. You didn't mention how high the 6004 settings(typically about a +/- 12dB range) to the mains were.
The amount of power used from the PM16 for a given volume level of the main speakers is the same regardless of whether you get there by turning the PM16 volume control high or by turning the main speaker trim settings on the 6004 high.
The "huge" volume difference you described is a bit unusual, although the center speaker might be more sensitive than the mains and/or the amplifier in the 6004 might have a higher fixed gain than that of the PM16.
