Don't confuse military with civilan math. Those military guys need math because they have to do some technical calculations, one Navy pilot explained to me at an airshow he maid it sound complicated. In civilian flying, you only need a certain type of math, basic 4th grade algebra. I would say Physics is more important. Don't worry too much about math.