Just because you like flying doesn't mean you have to do it for a career.
Why can't you just enjoy flying as a hobby or as a means to a particular purpose, like volunteering or as part of a business?

On the other hand, if you are just beginning this road - it might be a little early to thrown in the towel. I know someone who was ready to give back the puppy she adopted the first night at home. Now a few years later she could hardly imagine not having the dog.

Perspective is a finicky thing.