I'm in Norwalk CT just off I95. Early show times I can be at the parking lot in EWR in 50 minutes. During rush hour I give myself 90 and have never had a problem in 8 years. Getting home from EWR is usually 75 minutes but in the summer on Thursday-Saturday rush hour could be almost 2 hours if you land at the wrong time. Once you figure out all the back roads to get around traffic and accidents the drive is very easy.
The area is great for families, amazing public schools in Darien/New Canaan/Weston/Wilton. Housing prices in those towns are insane though. Train to NYC is about 50 minutes. Long Island sound is great if you're a boater and good skiing in VT is 3 hours north.