Financially it makes more sense to invest properly than to pay off the house as you have pointed out. However from a financial security standpoint, owning* your own home outright is a great safety net. So it depends on what you're seeking... financial security, or greater amounts of wealth. That's a personal decision.
(*you don't really own your own home due to property taxes, you just least it from the government like serfs)