The Goggle time is very optimist since it never gets stuck behind a slow moving truck on a curvy road or makes any provision for stopping and just looking. Given your limited time, I would skip RMNP since there is nothing there that you will not see somewhere else. The drive through the Glenwood Canyon is awesome and you need to hit a couple of the pull outs. Or better yet, rent bikes and ride the length of the canyon from Glenwood Springs. The drive into Aspen via Independence Pass (Leadville) is as good as it gets. Hike or bike the Mineral Belt trail in Leadville for a unique exposure to the early mining days of Colorado. Try to believe that people once lived there. The drive from Aspen to Denver via I-70 on a good day is 4.5 hours and much worse on a Sunday afternoon.

Last weekend, the colors were just beginning in Winter Park. I wouldn't let the leaf change dictate your trip.

I wouldn't disagree with anything fmpden and PeaceOut are saying. The scenery through the entire trip to Aspen and back, doing the two different routes is just stunning and full of things to see and do.
ENJOY.

Thank you everyone! You've given me a lot to think about. I'll try to give a trip report when we return. Rocky Mtn High here we come!