Blame it on West Point
How it started: Graduates of military academies had a four-year degree, so other officers needed one too. Military fixed-wing pilot training was mostly limited to officers, so almost all the military pilots had degrees (and these days, having "only" a bachelors degree may raise eyebrows at some promotion boards). The airlines used to hire almost exclusively from the military, so any civilian applicants needed a degree to be competitive. Those who did the hiring, of course, had degrees themselves. Perhaps the rationale has been that a degree helps with the non-flying aspects of the job, and that's just become "conventional wisdom". In any event, applicants have little choice but to show up with the credentials the companies want.