An old GF of mine had a yeast infection once and believe me, it's nothing like the "fish smell" ... It's unbearable. You are right that it is most likely a hygiene issue, but also people do have different chemistry's.
I agree that in general, shaved does not have that "musky" smell of some that are unshaved.
I have to admit that a little aroma is nice, in low doses ( bad pun ) as opposed to squeaky clean.