There are physical limitations which can prevent one from performing necessary duties. For example, a "little person" who stood 3ft tall would be unable to perform their duties and therefore would be unemployable.

Obviously someone who is 6'9 would not fall outside the envelope of being able to perform their function, but there's no problem with discriminating against someone who can't do the job.