My wife and I are planning to spend 3 days in Dublin around New Year.

Would any one have sugestions on "must do" in Dublin?

first, slow down you are in ireland
second make sure you have evening meals booked as everyone will be in party mode and the place fills up
1) Trinity college
2) the guiness tour
3) Dublin art gallery (forget its name) but just north of the main war memorial
4) dublin prison complex

Dublin Castle
A host of wonderful museums
Theater in the evening (a local if possible - Shaw or Wilde etc)

We thought the tour of Kilmainham Goal was one of the highlights of our trip. We also went on the Historical Insights walking tour and it was very interesting and well done.

If you're into nature and history, then a visit to Glendalough could be a highlight of your trip. There are daily buses from Dublin and at this time of year it is unlikely to be heaving with tourists.
http://www.jeremytaylor.eu/glendalough_ireland.htm

Do your research in advance - some places might be closed because of the Christmas season. The Book of Kells at Trinity College will not be open over Christmas until January 3rd as far as I know.

The Dublin Literary Pub Crawl is very popular and very good.

A visit to the Abbey or Gate theatres if you're culturally minded is recommended.

I'm a big fan of the Chester Beatty Library in the Grounds of Dublin Castle. Entry is free and they have an excellent cafe.