You can wander around all you need to see in 3 days. Much of the sites are centrally located. If you feel the need to visit the Cu Chi Tunnels/Cao Dai Temple, then perhaps 1 extra day.

I wouldn't call Saigon modern, but they're trying....

Bisbeee is the expert here and I agree that 3 days is good, with one more for daytrips. As I've said before, Saigon is one of my favorite Asian cities but actual "sights" are thin on the ground.

Take a look at this blog:

http://www.noodlepie.com/blog/bun_moc/index.html