Mystic, Stonongton, Essex. I would not stay in New Haven. RI is nice too.

Mystic Connecticut is wonderful for families. We loved the Mystic aquarium. There is also a nice little town to walk through along the seaport. Mystic seaport is another attraction that has hands on exploring. We stayedd at the Inn at Mystic. It had an outdoor pool, nice breakfast and sits up on a hill overlooking the harbor

Mystic also has plenty of hotels and motels, some cute shops and good places to eat. However, it's much closer to Boston than to Delaware! Also, it's a significant detour to go east of New Haven instead of heading north there to Boston, so only do it if you're really interested in staying in Mystic.

New Haven is a fairly large, industrial city, though it has its charms: the center of town, Yale campus, and a good range of food options, it's most famous being the local pizza.

...If you're interested in the pizza but not in stopping in New Haven, the local Frank Pepe's has expanded to a number of locations throughout the state. Very simple menu: all pizza and one salad LOL. We like going to the Fairfield one because it's right off the highway. And if it's too crowded, we go to a hot dog place nearby! New Haven's other famous pizza is called Modern.
pepespizzeria.com

In Mystic, we have a tradition of eating fried clams and lobster rolls at least once a summer at Seaview. Next door, the tiny Kitchen Little is awesome if you're in the area for breakfast - way better than the chain places up the road!
kitchenlittle.org

As for your original question, how about Greenwich or Westport? The latter especially has the prerequisite cute shops and restaurants, and I believe their beach is more accessible than the one in Greenwich...

As ggreen writes, you will add 30 to 40 minutes to your total driving time by "detouring" to Mystic, but I think it is by far the best family stop. New Haven has Yale, and that is important only if you are bent on indoctrinating your children to attend that excellent university. You might be in Fairfield around lunchtime and if so greens rec of Pepe's is good. The hotdog place is great, but the wild topping options might intimidate the kids! http://www.superduperweenie.com/

Greenwich and Westport are affluent, charming (sort of -- they've become more or less open air malls) suburban towns but have little of interest for a family. Nice places to live, not so great to visit. (I know because I lived exactly between them for 16 years.) Madison Ct is another possibility, but really, push on to Mystic -- it's only another hour!

In Mystic ggreen's rec of the the Seaview is right on, Sea Swirl, on the other side of town is also good, though not as scenic. There is another called The Cove which I want to try.

Mystic is the obvious choice for a family - between the aquarium and the seaport museum - s well as the beach there will be plenty to do. New Haven is a business/industrial city with really nothing for small kids,

If your trip is this weekend, don't go anywhere near New Haven - Yale graduation weekend, with the actual ceremony being on Monday.

Thanks, Fra, for confirming my eatery recs! My nephew's eyes were as big as saucers the time we took him to Super Duper Weenie. For what is in essence a limited menu, it took him longer to decide what to order than for the order to arrive and be consumed LOL.

And I agree about Greenwich and Westport. I was just trying to think of someplace coastal before reaching New Haven... I'd definitely choose Mystic over Madison - and I agree if you're going to go to the one, better to carry on to the other. (Though there's a big fried clam place in Madison, too. The one time we went - because we wouldn't reach Mystic before closing time - it seemed like the entire town of Madison was in that back yard!)

Thank you all for your fantastic recommendations. Because Mystic is such a long drive , I'm thinking that we'll stop in Westport . I found a place with trails called Earthplace, any reviews on whether it would be a good place to hang out for a few hours with the kids before we continue on to Mystic for the evening and following day would be appreciated .

You could always stop by Keith Richards place in the next town over and send him my love.

I like this place in Stonington

http://www.dogwatchcafe.com/

Westporter born and bred here....you'll love Earthplace! My nephew goes there all the time. Downtown Westport has become very mainstream but it still is charming and a cute Main Street nonetheless. You could also take them to play at Compo Beach although it could be hard to park there if you don't have a beach sticker (may have to pay more). The annual Yankee Doodle Fair is from June 14-17 and has lots of kids rides.