No, it's much, much simpler than that.

When you are IFR, you are on a clearance the entire time you are in controlled airspace. Class E, D, C, B or A. Doesn't matter what airspace. You are on a clearance.

Compare that to VFR flight. There are no such things as airspace clearances for VFR flight in controlled airspace (except maybe "special" VFR) - even Class C is just a communication requirement - with one exception: Class B.

Flight in Class B requires a clearance. The difference between VFR and IFR is simply that VFR you don't have one until they give it to you. IFR you have one all the time you are in controlled airspace (unless of course your whole flight is illegal),