I myself, as well as several successful pilots I know, did it slowly while still going to college. I'd put school as a priority over flying, though. Flying will always be there. You're only young once and going to school can be a pretty good time. Make the best of it.