Our cruise ship will be docking in both of these ports and I'd really appreciate help in planning our itineraries. What's a must see in each of these ports? Can we do it on our own or do we need a city tour?

Thanks.

Akureyri is a tiny place. Not much to do or see. Depends how much time you have. If you are not pressed, try to rent a car, skip Akureyri and go to Myvatn.
Rejkayavik is larger, quite pleasant, has an interesting church, port nice narrow shopping street etc. Again, depends on how much you have. You could go on a tour to Gulfoss, Geysir anf Thingvellir (many bus tours are offered, I thing they call it the Golden Circle Tour). This is a nice trip that will give you a tiny taste of Iceland. You could also join a tour to the Blue Lagoon. I personally do not care much for this artificial pool.

We spent three nights in Akureyri. It's a great town to walk around. There is the world's most northerly botanic garden (you actually can walk through trees there, which is unusual for Iceland). The cemetery was fascinating.

Also two very nice churches, one is a very very small Catholic church. We toured the Nonni (sp?) museum. I guess we did just about everything you can do in town! It was a relaxing place to visit -- Akureyri is Iceland's second biggest city, but it is really a very nice small town.

One day we took the bus to the north and took the ferry to Grimsey Island (north of Arctic Circle). That's a great place but I'm sure you don't have enough time to go there.

The swimming pool at Akureyri is very fun!

Swimming pools everywhere in Iceland are a lot of fun.
I agree with alfisol's comments about Akureri. But, do you want to see another small town, as sweet as it might be, or do you want to have a glance (even a short one) of what Iceland is about?