I've been driving from Cape Cod to EWR for three years. At 4+ hours, its just doable. I actually have to leave the house earlier to go to BOS to fly down.
The Boston commute can be tough. The schedule is sparse in the middle of the day, the flights can be oversold for weeks, and there are lots of guys on the jumpseat. If you're late getting back you may be in for an unplanned overnight.
So, yeah. Its doable.