I knew about the FAA guy sitting in beforehand. I feel you have the right to say no to the check ride, but like MikeInTX said, you will have to part with your dough. The DE should tell you if he is being examined when you schedule.
If the FAA guy just happens to show up then you're screwed. They can do it whenever they want.