Exactly, safely getting the people/cargo to the destination. He is not responsible for stuffing the cockpit with anyone who wants a lift. The Captain can, as a courtesy and at his discretion, allow certain others admission to the flight deck even though they are not responsible nor helpful in the safe operation of the flight. If, for any reason, he/she chooses not to extend that courtesy, the issue is closed. Why is that so hard for you to understand?
BTW Glen, I see you have been an Airman since 1976, are a mechanic and are thinking about "moving up" to the flight deck. Assuming all these things you've said are true, you got a bit of a late start don't you think?