In a nut shell the mode s transponders communicate with one another. The then calculate the poszibility of an impact and then count how far away the impact is.
Just as a correction above...mode S transponders aren't required...TCAS works with Mode-C as well. The TCAS system figures out collision hazards and not the transponders. If one aircraft has TCAS and the other doesn't the aircraft with TCAS will still get an RA.