Both Aschaffenburg and Mespelbrunn are surely worthwhile. In addition to those, check out:

Büdingen
Gelnhausen, witht he ruins of Barbarossa's imperial palace and the old town
Seligenstadt
Miltenberg
Amorbach
Wertheim
Michelstadt
Ronneburg castle
Wilhelmsbad palace and park in a suburb of Hanau, and the suburbs of Kesselstadt with Philippsruhe palace, and Steinheim (otherwise Hanau is a promising candidate for the title of Ugliest City In Germany)

Thanks a lot, quokka for the quick response... I will look into each of those towns for sure... We are planning Aschaffenburg for today evening and go to the Mespelbrunn castle tomorrow morning at 9 AM... Since the castle closes at 5 PM, we cannot go today... So tomorrow morning will go as early as possible to catch some soft light as I am into photography and do not like afternoon light...

Now for the rest of the day tomorrow, my work just got cancelled so I have the whole day... Is Heidelberg worth a day trip? Or should we stick to the small towns you mention above? We can now drive about 1-1.5 hrs in any direction...

Since you plan to go to Mespelbrunn first, I would not combine with Heidelberg wich is an entirely different direction. Do I assume correctly that you are going by car? Mespelbrunn could, for example, be easily combined with the old town of Miltenberg (think half-timbered houses and everything) a bit further South on the Main, and Amorbach with its baroque abbey. This route takes you on small country roads through the forests of the Spessart hills - beware of the robbers, hehe (there is that famous German musical and movie, "The Spessart Inn").
If you still haven't had enough after Miltenberg and/or Amorbach, you could take the route through the Odenwald hills back via Michelstadt, and there is also a fine medieval castle along the way, Burg Breuberg.
This route takes you through a lot of different landscapes and to five off the beaten path but worthwhile destinations but total driving distance equals about one way to Heidelberg. You'll find all the picture-book images of "old Germany", including half-timbered houses and cobblestone and castles and forests and everything, on this trip!