CASS is exactly what it is ...Cockpit Access Security System. There is no requirement to clear you with CASS unless you need to actually occupy the JS.
Had the same exact thing happen to me, played it cool but insisted that I speak with the Captain and rode in the back no problem. Sounds like yet another guy screwed by a gate agent that doesn't know the rules, or worse thinks that they know them all and are wrong!