ChecklistMonkey, You may want to rethink your thoughts about the Captain not being your boss. FOM 10.2.2.1 spells it out. He can have you or flight attendants removed from a flight (I have seen F/As removed a couple of times). The first bullet point in that section gives the Captain that authority.