I understand your frustration with part 61 training and the long process that instructors can make of it. I left for 141 training to get all my flight training done quickly and with loans. Our situations were different.
You have a good job, I didn't.
You have an airplane to fly, I didn't.
You have instructors slowing you down. I really had money slowing me down, but my instructor was part-time.
My advice:
Keep your job.
Fly the Arrow.
Get a new instructor or more than one. You're the boss, fire your instructor and find a new one. The least you should do is talk to him about it.
Get your instrument, then commercial, then multi-eninge, then all your CFI ratings. Worry about building twin time later.
Have fun. Good luck.