Domestically?

Small towns (quick van rides): Downtown layovers are AOK.

Major cities (long van rides): No thanks. Life is too short to spend 1-3 hours of a layover sitting in traffic in a van.

All American cities increasingly look the same anyway. Downtown safety is becoming a thing again. (aka "Murder-apolis". So much for Minnesota Nice.)

Internationally?

Yeah, better go downtown. Why fly international if you're not going to see some of the sights?

Your experience may differ.