The reason it's in such demand is because there is high pay for professionals and there is WORLDWIDE demand for housing which is limited due to geography and urban planning. Every immigrant who gets a job in silicon valley buys a house. Then he buys one for parents and/or in-laws, then one for rental. Then people from out of state and even far away as China buy up real estate for rental. 50% of housing is rental. Loads of houses have two or more families living in them. Public transportation is pretty available here, and the light rails are not patrolled for fare jumpers.

On top of all that, the weather is great, there are lots of good schools, and tons of low paid service jobs for the less educated. Kids don't leave home and just become part of multi-generational families. They can spend their money on stuff, because buying a home is simply impossible. During the downturn, Million dollar homes went down to 750, 500K homes dropped to 350-if you could find them. Outside of the bay whole neighborhoods were empty in places like Manteca, but NOT for sale. The banks got bail outs so they just sat on the homes, refusing to release them.

The key to living in the bay is live near where you work, or work late at night. Rush hour is 6am-10am, 2:30-7:30, but traffic jams occuring at any time of the day. The city planners are actually REDUCING lanes in many cities in order to put in (unprotected) bike lanes in which people don't feel safe riding in crazy rush hour traffic.

Blame silicon valley.