Copperhed,

I was in a situation such as yours in the mid-'90s. I was frustrated, fed up, envious of others' accomplishments (and finances) all of which led me to be totally turned off of aviation.

I got my "more stable" job, made "good money" but after some time I realized that I'm simply compensating myself for being unhappy. Sure, there've been bright spots along the way but I still look up when I hear a plane and look out the terminal window when I travel wondering.
All of the "what if's...", especially when I find out about other people I knew having established themselves in a good aviation career, could come back at you. You may find yourself asking as to why you couldn't have tried harder to have both.

I'm not suggesting that you're not making the right decision. I know nothing about you.

I'm just saying that I hope you're not making life-affecting decisions based on your current circumstances (which could be temporary).

It's a tough call.
Good luck.