Everything here is opinion and common-sense according to Jeffery, so it could be WAY off.
That said, I believe it's a bit of both.
A skimmer rated at 75 gallons on a 400 gallon tank is not going to get enough water through it for it to make a decent dent even if there was a TINY bioload. Basically the bioload would need to build up dramatically in order to get some of it pulled off in the underpowered skimmer.
Likewise, a skimmer rated ridiculously high in comparison to the bioload will not function properly. You will have a terrible time getting it to skim right without a decent bioload.
I have experienced the second scenario. A 100 gallon rated skimmer with a pump upgrade, on a 60 gallon tank that was very lightly stocked. I couldn't get it to skim for crap until I doubled my bioload and then it worked right.
Overall, it is my opinion and experience that a skimmer needs to be sized for both water volume and bioload with bioload being a bit more important than water volume.