Remember that airspace rules are generally designed to benefit IFR traffic. By setting up special VFR, VFR only pilots are afforded a way to safely land in the event of weather or other conditions that would prevent them from returning to their field. However, the effect is to make a controller aware of that traffic, so that IFR pilots don't pop out of an approach only to find some poor Cessna lined up on their final.