OP,

My advice is wether you start now or not (with flight training), your priority is college. Don’t take a break, as it slows your momentum and could become a permanent break. Make sure you keep up a good GPA. If you’re a physics major, it’s going to be pretty difficult to fly much.