There are two interests here: protecting yourself professionally, and being personally, and justifiably, offended.

In my opinion, the first is most important. I would pull out my phone, call my CP, and not so subtly state that it "had been suggested, without cause, that I might be intoxicated. She's right here, would you like to speak with her? Shall we cancel that flight while I get a breathalyzer?" You might not feel compelled to go that far, but at the absolute minimum, I'd ask for guidance from the company. That way they've vetted you, and can't easily subsuquently subject you to ridiculous "disciplinary action". Handled smoothly, this would likely cower the passenger I'll refer to as "Madame Bovine."

If I were working the flight, I'd imagine the worst-case scenario, say a tire blows or my Pratt & Whitneys combine with goose migration patterns for a bad day. Then it's like an off-duty cop at a bar that kills a mugger: You're automatically suspect, an easy target, and it doesn't matter how right you are. It doesn't have to be true to ruin your career, especially when several hours later "of course he blew a .000" I think if you're about to fly, in this case you have to take measures that might seem Draconian to anyone whose career is not on the line. Better safe than really, really sorry.

A few years ago, a friend of mine had a passenger "joke" while boarding that the crew should take a sobriety test. So they did what I would've done, and initially asked their chief pilot. They were told that to go get tested. However, management backpedalled like a 3 year-old in high-speed tricycle reverse when they realized what would happen to the flight, and what it would cost. They got tested anyway.