Male/female size differences would have evolved prior to humans as a recognizable species evolving - and the fossil record of pre-humans supports this.

Humans have never self-selected for physical fitness with any regularity, throughout the historical record. We primarily mate for social reasons.