Part of it is that our aviation system, through accidents of history, evolved around pilots who were pretty smart and capable. The system now depends on that, although technology and automation has taken the edge off of that just a tiny bit. But airlines still absolutely rely on the pilots to be the Captain of the Ship when it comes to flight ops safety, even if they don't like to admit it.
So people who met the standards to be pilots for the better employers (who also want some people skills, whether airline or corporate) have other options in life. Since you need them to spend a lot of time away from home, you have to pay them to incentivize that. If I could make a lot more in other fields AND be home every night, I'd have trouble justifying aviation to the family.
Also pilot unions help increase and stabilize wages. Airline pilots are not as replaceable as mechanics, and pilot unions allow limited or zero outsourcing. During a strike, an airline can't hire and train thousands of airline pilots over night... takes too long, there's not enough pilots out there, and the sim capacity simply doesn't exist. Mx can be outsourced more readily and quickly.