Oh man...been there done THAT. When it happened to me I just looked at the guy and said "listen, I get paid the same either way, do you just want to fly all the legs so you get to fly them your way?" He responded, "No, I'd still like you to fly your legs." So I responded "Well I'm qualified to operate this aircraft so unless I am breaking a reg or a limitation, not following policy or I am endangering your life or others then I'd like to operate the aircraft my way. If thats a problem for you then I would prefer you fly all the legs so theres not a conflict".

In your case with his response about challenging his authority I would tell him that you're not challenging/questioning anything but you ARE telling him that the environment he creates in the cockpit is not the most positive and safe. If he still has an issue with you flying the airplane then it might be worth talking to a member of your professional standards committee. Chances are this is not the first time another pilot has had an issue with him so you can take comfort in that.