Better to graduate with ‘basket weaving’ than flunk out of the physics program, just saying. Plus you can likely get through sooner.
I have a relative with no degree, has accomplished more than 90% with a degree. He owns a major business, pipelines, land clearing, earth moving etc..
Of course, nothing wrong with backup plans.