I only have time for one day trip and have narrowed it down to either Red Rock Canyon National Conservation Area or Hoover Dam. Which would you choose? This would be at the beginning of October.

I think this has to be pretty much personal preference. Just to complicate the issue, Valley of Fire also ought to be considered.
The dam has technology and the view of Lake Mead going for it.
Red Rocks has the unusual landscape.
Valley of Fire has unique landscape and, if you come in or out of the east entrance, you will be driving past Lake Mead.

My personal preference would be Valley of Fire.

I also think Valley of Fire should be considered; I have been to Hoover Dam and Red Rocks, and while both have their very different "beauty", it sounds like Valley of Fire would be spectacular and unique.

If you get going early, pack a lunch, head north on 15 to Valley of fire, spend some time there, check out mouse's tank and look for petroglyphs on the left as you go in, check the ccc cabins, eat lunch, then drive through and loop along lake meade and visit Hoover Dam by mid afternoon, take the dam tour and then back to vegas for supper.

I would put them in this order:

Valley of Fire
Hoover Dam
Red Rock park