I use it when we have a light morning fog and it's burning off, but not quite "VFR" conditions yet. I've been driving in, and can see the sun, but not at the airport.

When I was at Purdue, the river surrounds the airport and most mornings in the fall were horrible, but it would be CAVU by 9-10. You could easily depart to the north when it was "1-2sm" and instantly be severe clear. We couldn't use it though.

My current job, I do a lot of test flights on planes coming out of Annual or Avionics work. Depending on the scenario and type of aircraft, I will occasionally use SVFR to get out and do what I need to do.

So in conclusion, SVFR should be used by someone that
A. Is extremely familiar with the airspace and surrounding terrian/obstacles
B. Knows the weather is improving
C. Has actual instrument time and experience(this is a just incase it's worse than reported)