My experience is dated (about 35 years old); however, the guys I saw trying to leave the Infantry to become an ARMY pilot were treated as worse than traitors by their superiors and peers. I hope it has changed.

(FWIW, I was commissioned in the Infantry with follow on orders to flight school in 1982, before there was an Aviation Branch. I was always singled out for "Special Treatment" going through IOBC [carry the M-60 on road marches, carry the mortar base plate, weekend duty, you name it.])