Dear Wannabe ...
First, I hope that 24 isn't too old; if it is, I'm ancient! In fact, I should probably be dead!
You're not behind the power curve, but, you'll never be in front of the power curve unless you go for it. If you do "go for it", don't half step and go in with your eyes wide open.
Lots of sage advice here based of lots of different experiences. I agree with some and disagree with others; however, if any of the words here have caused you to contemplate what you really want and/or need, the advice shared here has been good. As many have said previously, the only person who can make the "is it worth it" decision is you!
You're 24. Where do you want to be 30-40 years from now. Seek out those guys and find out how they got there. There are a lot of great professional pilot careers besides spending your last years in the left seat of a 777 going trans Atlantic. There are a ton of different paths to open the doors to aviation possibilities. Have a plan! ... and contingency plans!
Should you choose to pursue aviation, start working on your degree yesterday and do it concurrently with aviation. I presume that you have GI Bill eligibility; use every penny (whether you fly or not)!
Some here have recommended that if you do want to fly, you should get a non aviation related degree. I guess that's good advice if you consider diversifying your options. I've found, however, that having a degree in anything is better than not having a degree at all. A few years after you get a degree, or, your second or third job after getting a degree will care little about what you majored in ... aviation or otherwise. (Not that my experiences are better or worse than anyone else's; just different) Years after getting my BA, I went back to school and got a masters. Although I've never worked in a field related to my course of study, having my masters did make the difference between getting an interview or not getting an interview several times ... for what that's worth.
Bottom line ... pursuing aviation can be expensive. But so is pursuing a career as a lawyer or a doctor. Only you can make the "is it worth it decision". All occupations worth pursuing entail sacrafice of some sort or another. In whatever you decide, I wish you the very best!