The toughest part about starting older is sitting right seat to a 25 year old captain, even if he/she is a great captain. It requires some discipline and humility. But so does every aspect of this job.
If you love flying and are financially secure (cuz the pay is weak) give it a go.
We have a few at my company. And I once flew with a former Air Force pilot who didn't fly for 20 some odd years and was a school teacher. Then he decided that flying was what he wanted to do and at the age of about 47 decided to get back into it. He was a 55 year old captain when I flew with him.