True, you're a big boy (or girl?) and free to do what you want. If aviation is all you want to pursue by all means - go for it!

With respect to an aviation degree, what I hear some of the experienced pilots saying is diversify yourself. Of course, we all have colorful ways to say that

I view education and/or skills as a stock portfolio. As the saying goes, "don't put all your eggs in one basket", it's also true with education. If you were to loose your medical, how many jobs are actually out there (that you're qualified for) in aviation.

I also have an aviation/engineering degree, which filled a square on the pilot application. Beyond that, the [college aviation] education has been next to worthless. I say next to worthless only because it allowed me to fill a square on the grad school application which WAS NOT aviation related. If I had it to again, I'd pursue an undergrad degree in Finance or something more universally applicable.

Good Luck.