This is all much ado about nothing. The OP said they wanted "to spend a night in a nice small town". New Haven is NOT a nice small town. Thus, the answer to the OP's question is NO, don't detour there.

A 1-hour train ride to Stamford could get you into the taping of the Jerry Springer show.

I do not think of New Haven as a tourist town but I was surprised to see many on a brief trip a few years ago to visit old haunts and have pizza. Nice small towns like Branford and Madison are close to New Have but you would really need a car.