If you were cleared to join the localizer you should be able to continue the lateral track without approach clearance, but definitely you wouldn't be able to descent out of the last assigned altitude you received. And if you were not cleared to join the localizer, or intercept the xxx radial/course inbound (VOR/RNAV approach...do they still do VOR approaches these days??) then you probably wouldn't be cleared to follow the lateral track inbound either.
So it obviously depends on the clearance you have received prior to the FAF. If cleared to fly the lateral portion but not the approach, then fly the lateral, but you can't vacate the last assigned altitude. If given a heading but not cleared to fly the lateral track via a call to intercept the localizer, or fly the xxx course inbound, then you can't deviate from the last assigned heading or altitude.
Do you always need an approach clearance though? Not always. Let say you are operating IFR and on vectors downwind for ILS 36 at uncontrolled airport Kxxx which is your filed destination. ILS36 has a FAF of DONNY and FAF altitude of 1500 feet, and an additional fix of JONNY 5 miles outside DONNY at 2000 feet. Your last assigned altitude was 3000 feet, and ATC gives you, "PROCEED DIRECT JONNY, CRUISE 2000". With this clearance you can maintain the block 2000-3000 until established on a segment of the approach at your destination, then descent via the ILS36; no approach clearance is required (although I would probably still verify it...controllers may not understand the CRUISE CLEARANCE any better then many pilots do). (Ref: Pilot/Controller Glossary page 621)
The interesting aspect of your situation is the combination of: (a) Class D airspace, (b) that you were not talking with the local controller prior to entering D airspace, and (c) that you were operating VFR at the time. Probably best not to enter Class D without talking to the Local Controller, so good idea to discontinue the approach and remain clear of D airspace, even if you had received clearance to fly the lateral track. One option available was, prior to entering D airspace, and because you were VFR (assuming you weren't in class B or C airspace of course), you could have just told the Approach controller you were switching Tower as you approached the D boundary, and coordinated with the Local to continue your approach.
However, what if you were: (a) talking with the local controller, (b) didn't have approach clearance, but (c) had been cleared to continue for the runway (continue inbound, expect RWY 36). You could have flown the lateral track and descended as required to land RWY 36 since you were operating under VFR. You couldn't land RWY 36 without clearance, but you can continue as required to land 36 until either the local called your go around, or you thought it unsafe to continue any further without landing clearance.
Welcome to the "lawyering" side of being a pilot. Many lawyers deal less frequently and with far less complicated "regulations" then we pilots do. RDA