I will be visiting a friend in Luxembourg and thought I’d tag on about 10 days of visiting two (maybe three) mid-size to smallish towns that have charming Christmas markets. But where to go? Colmar? Koblenz? Mainz? Maybe even Strasbourg? As you can see, I am clueless. While I love big cities, I think for this trip I want to focus on the small and romantic at Christmastime. Where have you been that you might recommend?

Trier has nice Christmas markets and is an interesting, historic small city in its own right. It's an easy half-day trip from Luxembourg.
Farther away, Valkenburg is pretty unique (it has Christmas markets in large former quarries underground, and is known as the Christmas town of the Netherlands and for having apparently the only still-standing castle ruins in the country). Aachen is quite close and is beautiful, historic and with very enjoyable markets; you could combine the too.
Siegburg, near Cologne, has a medieval-themed Christmas market with torches, jugglers, traditional handicrafts and whatnot; I've never been, but it sounds picturesque if a bit kitschy.

I loved Colmar at Christmas!

If you want something really authentic, take a look at the small towns in the German Erzgebirge (Annaberg-Buchholz, Schneeberg, Seiffen, etc).

Thinking of markets that I've been to that are close to Luxembourg - Luxembourg itself has a smallish market that is nice, and it's very pretty at Christmas time. Aachen is fantastic, although the market isn't so small, but plus points for Aachen. Everyone comes out on the evening of the 23rd and fills the streets. And Paderborn has several markets together that are really good also, although I'd never call Paderborn tourist-fodder. Mainz is mid-sized, I think you would enjoy Mainz. There are a lot of other reasons to visit Mainz also (museums, Altstadt).

Also nearby is Düsseldorf, which has the biggest markets I've been to - kilometres stretching from the inner city to the river and ending with a ferris wheel. Lacking that small-town appeal you crave, though, but a lot of fun. Münster seems to get a lot of Dutch tourists, I liked Münster too, and it's a very pretty city. Of all of those I think I would prioritise Aachen. It is pretty, very tourist-worthy and has a really good market, but the rest are nice also.

Lavandula

Wow, thank you, Melinq8, lavandula, and mbrevitas. This advice is already very helpful!