It depends on your interests. With only 4 days in Paris I wouldn;t - since the gardens and fountains (not at their best or on in Nov) are a big reason to go.

I wold:

Make a list of your must sees - and see if you have n extra day

Consider the weather - if it's bad Versailles will not be worth it

It is totally dependent on your interests. If you are a devotee of the Sun King and the history of the period, of course you should go. If you don't know squat about the history of the period, why would you bother except to gawk at the castle and get some minor injection of historical info from a hugely influential piece of architecture or two?

Agree with consider the weather.

I enjoyed wandering the streets where the courtiers lived and seeing Le Jardin du Roi, a huge walled vegetable garden in the process of reconstruction with old varieties of fruit and veg. I wonder if it was completed?

I've been to Paris twice and both time included a trip to Versailles. To be quite frank, I'd recommend Versailles over travelling up to the Montmartre area and Sacre Coeur every time. Depending upon where you are staying in Paris, the trip times from the two are about 12 minutes different.

As for weather, one trip was in February, the other in Mid March. The gardens were a wonderful part of the March trip, and Marie's Hameau is a must see in my opinion. I agree with others here that if the gardens are an integral part of your plans at Versailles, then the weather will be the most important factor.

apersuader makes an excellent point IMO - the travel time to parts of Paris may almost be as long as to Versailles, and the things you spend nearly as much time to travel to inside Paris may well not be as awesome or as enjoyable to you as splendorous Versailles - the world's most famous palace.

Versailles is really an integral part of Paris in this thinking.