Correct me if I'm wrong, but isn't one of the reasons that pilots and FAs wear uniforms is in the case of an emergency situation? The thought process being that passengers would be much more inclined to follow the instructions of someone who looks like they're in charge and know what they're doing. If pilots were to be wearing something business casual, then that pilot yelling at you to not take a certain exit could just as well be some guy traveling to a business meeting.

I'm not sure if companies actually take that into account when designing their uniforms, though since it's not a very likely situation to occur.