I used to feel bad about getting an uber for a 4 mile drive to LAX, but after paying $50-$90 to go from LAX to ONT/SNA and having the drivers complain the entire ride about how it was taking them in the opposite direction THEY wanted, I stopped caring. Like everything in life, nothing is perfect. If they hate it so much, they should just quit. If they don't complain about the drive, I give them a nice tip.
On the other hand, I had a driver who picked me up for an 8 minute drive to LAX who loved it because if he did enough rides by the end of the day he would get a bonus, so he loved the FA and pilot airport runs because he could rack up a ton in a short amount of time.