I thought i do the day trip to venice from milan.

That doesn't work really, unless you want to spend 5-6 hours on a train round trip, to have only a few hours mid-day in Venice. I agree if you can't give it at least 2-3 days, I would drop it (even though to me it is the best thing on your itinerary!)