I plan to do a day trip from Madrid to Segovia. I want to see the main sites and return to Madrid without too much lingering. How many hours would this take as I want to book a return ticket.

You'll want to set aside enough time for lunch in Segovia, say 8 hours, maybe a little more if you'd like to enjoy a relaxing lunch, Spanish style.

We did not have a long lunch,
so about 5 hours covered it.
Check when the bus leaves for the train station...we did not have time to wait for the bus , took a
taxi to make the train.

Take the comfortable and cheaper bus rather than the train-The bus drops you off in Segovia,a short walk down to the aqueduct!The train station is further out and you need to then take a taxi or wait for the bus!
Get an early start-around 9AM to beat the crowds, walk up to Cathedral and then the castle.On the way back you should be ready for lunch and then the bus back to Madrid.5-6 hours should be enough time and then you can continue on to do things in Madrid!

The buses tend to coincide with the fast train because Segovia's AVE station is separate from its train station for non-AVE trains.

You can do what you need in 5 hours, max. Segovia is small and there are better day trips in the world.

Toledo and Segovia are my personal favorites for day trips from Madrid - but I haven't yet visited Cuenca and it's on the top of my "to do" list. . If you plan to enjoy lunch in Segovia I'd budget 6-8 hours total. I've actually spends couple of night in Segovia twice in lieu of a daytrip and it wa very enjoyable.