Gate agent and supervisor were both wrong.
Quoting our manual:
Jumpseat riders (both cockpit and cabin) shall be in uniform or casual business attire ...
If this happens to you again politely ask the agent to allow you to speak with any cockpit crewmember so that they can reference the above given information and correct the agent.