You need a really strong support network in place to make it happen and not ruin your kids or your marriage.
With the way things are going these days with hiring, you could always take time off while the kids are young and hop back into it several years down the road.
If you want it all, something has to give. Just make sure you don't give up the wrong parts. Good luck.