I'm taking my dad on a day trip to dc to visit a couple museums on the mall. Wanted to take him to a nice lunch somewhere and was trying to stay away from museum restaurants. Looking for a place that is unique or has some charm.

The Mall is more than a mile long and there's nothing really close. The restaurants at the National Gallery of Art and the Smithsonian's American Indian museum are probably the best choices without leaving the Mall, but they are both at the eastern end (4th Street).

If you want to make the most of your Mall time, eating in one of the museums would be best. If you want to spend 90 minutes or more getting lunch away from the Mall, go up 7th street about 4 blocks and find a rich assortment. Jaleo for Spanish tapas is my favorite in that area, but there are many others around the Gallery Place Metro (7th and F area).
Going to this area would add about a mile to your walking total for the day but you'd have a lot of options.

You can't beat the food at the Native American museum - it is better than most places in Penn Quarter. It is not cheap (not that any of the museums are) but it is good!

Another vote for Jaleo. Just a wonderful experience overall - good service, food, and atmosphere.

In the West Wing of the National Gallery there's also a very small garden cafe, not well known, that is more intimate and offers table service. Also, there's a nice Italian restaurant, Aria, which is in the Woodrow Wilson courtyard between the Reagan Bldg and the Old Post Office. From the mall at American History head north on 14th 1 block, and it's on your right.