We will be spending 4 days in Madrid but would like to use one day for a trip to either Toledo or Segovia.
Which do you suggest and why?

I have been to Toledo but not to Segovia so I can't compare the two from firsthand experience. I would visit Toledo if I were you. It's an ancient city filled with Arab, Christian and Jewish monuments. It's a very important city historically. We loved Toledo; we stayed one night and I wish we had a second night.

I have been to Toledo twice and Segovia once.
My overnight in Toledo stands out as one of the most interesting, mysterious places I have been and the cathedral is absolutely splendid, one of the great ones. Toledo has amazing history and is an example of cultures and religions that worked together and functioned peacefully. I know it would mean changing hotels, but staying overnight in Toledo, walking the warren of tiny streets in quiet darkness, is magical. Ride the tram around for views of Toledo from the surrounding hills. I did Toledo once as a day trip and it was much less satisfying. I would easily give up time in Madrid for a night or two in Toledo. For the best experience, spend at least one night there.

Segovia is very pretty with a lovely main plaza, great views of the surrounding countryside, an incredible Roman aqueduct, and the castle. There are also plenty of restaurants. It is nice for walking. It makes a good, but fairly long, day trip from Madrid. I think it would be enjoyable in the evening, but not so different from other small towns/cities in Spain.

Hope this help you decide.

Both are great, you can't go wrong, so it's really personal taste and time you want to spend. There is more to see in Toledo, of course.