QOL is EVERYTHING
We operate one airplane with two pilots, also flying about 400 hours per year.
Our owner, however, doesn't worry too much about contract pilots when we are using our company-standard PDOs. The PDOs are part of our compensation package and we lose them if we don't use them, so...they get used. If we take a vacation week right after the airplane gets out of a 3 week heavy check he'll probably have a little heartburn about that, but we don't get static about taking time off.
Your Chief Pilot should have the stones to set the owner straight if the boss makes snide comments like "you're on vacation every time you fly somewhere" and "you're just sitting on the road".
Additionally, something to look at is the cost of the crew RON for multiple days at a time vs. airlining back home. In most metros with airline service, the break-even can be anywhere from 2 to 5 days. This may save your owner some money while giving you more time at home...