how is the boat ride between st martin and st barts? is it rough?

Hard to give a definite answer. It depends on the time of year and the weather conditions on the day of travel. It sometimes can be a bit "lumpy". If you are prone to seasickness consider flying over. It will save you time and worry.

Going over can be rough, ie, "the vomit comit". Coming back no problem. There is a High Speed Ferry that is less violent, but even that...
The problem is the prevailing winds are on your quarter going over and a tailwind coming back.

i booked a 50 foot opencruiser on way over and a 40 ft on way back in dec. is this a good or bad time of year?

Again it depends, in late December you are likely to encounter the annual "Christmas Winds" which tend to bring on the swells. As the other poster said, going over is probably going to be rough, no matter what size boat you are in. Truth be told, there are lots of horror stories posted on this and other forums regarding the boat ride from SXM to SBH. Go by boat if you and other members of your party like adventure and are used to "open water" sailing. If you are relative neophytes or have only gone boating on calm waters you'll find the trip uncomfortable, at best.

Fly over to St. Bart's & sail back.

Fly if you can. We just took the ferry over a couple of weeks ago as we missed our flight to SBH. It was horrible! I have been on sailboats, ferries and other boats from SXM to SBH at all times of the year and I have always wanted to be sick. And I don't normally get sea sick. If you can catch WINAIR do it. GL