These "dumb rules" are one of the main reasons that, by some people's quality metric, the Valley is such a great place to live (if you can afford it, of course). Oakland, San Jose (at least the downtown area), and San Francisco all have pretty high density. Some people like cities, but I think most people would prefer, all else being equal, to live in Portola Valley or Woodside.
Being a nice place to live is a luxury that the rich enjoy and everyone else suffers for. Silicon Valley is the biggest tech hub in the world and home to billions of dollars worth of industry. Zoning should reflect that, rather than the whims of the lucky few who can afford to buy into the current inflated market.
Low density suburbia is what makes the Valley appealing? You can get that anywhere in America for a fraction of the price. What distinguishes the Valley is obviously its status as a tech hub. And most people may prefer to live in a mansion on the hills, surrounded by other rich people, just like most people would like to live in the Hearst Castle. But back in reality, people are choosing between a 1000 sq.ft. million dollar house in the Valley, and an hour long commute to something more affordable. The people who like the status quo are frequently those who bought a house in the 90's or earlier and saw a 500% return on their 10x leveraged investment, and don't want to see that price drop even a little. But even they complain about traffic.
by some people's quality metric, the Valley is such a great place to live
And, by other people's, it's a terrible place to live because you have to drive everywhere. As "The Soul of the Commuter" describes, driving makes people unhappier in almost every possible way: more socially isolated, fatter, and less connected to their communities: http://www.newyorker.com/reporting/2007/04/16/070416fa_fact_... .
