Not enough info;
Type of airplane?
Phase of flight?
Qualifications of the guy in the left seat? Is he single pilot qualified?
Personally I wouldn't have a problem with it if we're in cruise with nothing needing to be done such as fuel transfer or weather avoidance or anything else that requires one of us to be head down.
Want to be in VMC also in case a symbol generator/ display screen or ADC takes a dump while I'm out of my seat.
Pilots call but I think you may have overreacted a little here.
'Not right now Sir, we are working on a couple of things. I will give you a call in about 10-15 min. That OK for you? Thank you Sir'
That shows him you're not afraid to exercise your duties as PIC and gives you time to confer with the guy in the LHS. What did he have to say about it anyway?