I have a Gibson&Barnes with the liner and it is just too darn hot to wear anywhere but outside on the coldest days. I bring one of those lightweight fleece vest with me and wear it under the jacket if necessary AND the vest allows you to go incognito and works well on overnights.

By combining the vest with a waterproof collapsible windbreaker, you can take on most weather. You can wear the windbreaker over your leather on the walk-around in the rain.