With three days and a baby pick a home base Los Angeles, Disneyland, Santa Barbara or San Deigo. Forget day trips too much traffic. You will spend half your time in the car.
Have you considered flying from LA area to San Jose? If you stay in San Fransico you do not need a car. Parking is very expensive and traffic is as bad up there as down in LA.
What is your hotel budget? Do you want to visit museums, shop, play at the beach, be a tourist? Knowing that would help in making suggestions.