The only family input that counts is your spouse, and kids...If you are doing what you love to do, and your family is ok with it, what's the problem?!!
How many dads can coach a soccer team, volunteer at the kids' school, go on a lunch date with the wife or girlfriend (or wife AND girlfriend!!)...I have done all of that (except the girlfriend part!!), and so have many others.
I have learned to ignore input from anyone outside of my immediate family...when your in-laws or your parents start paying for a shelter and food and clothing and retirement $$$, and start paying for the kids' college, may be then they have some rights!!
And yes, it does get better with seniority...you have worked hard to get to a good place. Don't let this get you down, man, it'll make your life at home more stressful.
Good luck.