I'd suggest that you pick one: Pilot or SEAL and go for it.

To answer your question, I've met the rare pilot that was a SEAL but flying really doesn't fit into the job description on active duty. As Rick mentioned, if this is something you want - do it now.

http://www.seal.navy.mil/seal/osvetoverview.aspx