The driving is easy - presumably you are flying out of Dubrovnik rather than Split due to your choice of route.

In 2007 I drove from Trogir to Mostar, along the Adriatic Highway - details here...

http://www.fodors.com/community/europe/mostar---a-city-of-beauty-and-bullet-holes.cfm

Last year I drove from Split to Sarajevo using a different inland (and wonderfully scenic) route, details here...

http://www.fodors.com/community/europe/off-the-beaten-track-in-bosnia-and-herzegovina-juliat-explores.cfm

When renting your car in Croatia you will need to pre-order the cross-border card, which allows you to drive in Bosnia and Herzegovina, and is your insurance. I believe this is currently around 50 euros. I think you have to order this for Montenegro as well, though it seems to me it is one and the same card.

I've never had any trouble driving across the borders, but I have a British passport. Mostly I have been waved through - it was only the bored border guards at Kamensko who inspected all my papers being surprised to see a lone Englishwoman driving through. Usually my passport has not even been stamped.

I spent all of last week in Croatia and Mostar and Sarajevo and can tell you what we did.
We had a car throughout Croatia but didn't cross the border with it, but remember that you will need to get an international driver's license regardless in order to drive in Croatia. There seem to be two major routes in Croatia to drive: the autobahn (inland) or the coastal road (which we took all the way from Trieste, Italy to Dubrovnik.) The coastal route is beautiful but it takes a lot longer due to the windy roads on the side of the mountains.
In Dubrovnik, we stayed at Hotel Vis. It's in the Lapad bay a fwe minutes drive from the Old Town area and was a decent value for the amount we paid and breakfast buffet was included. We had an oceanview room. There was free parking here, something you'll need to consider with your rental. Staying in/ near the Old Town is not very feasible with a vehicle as you cannot take it into the Old Town and parking nearby is between 10-13 Kuna per hour (and you can never find a meter to pay it at!)
On a side note, a hotel is also a good option because aside from pasta and pizzerias, there are few actual restaurants to sit and eat in. Bakeries or "Pekaras" became our go-to for delicious meat filled local pastry as a nice change from pizza and pasta!)

Bosnia is lovely, we found the people to be nicer than in Croatia and the food is better (you will see everyone eating something called "Cevapcici"- a naan type pita filled to the brim with small beef sausages and served with a side of onions. It's good and a safe bet if you are a bit antsy about eating locally prepared meats.
Mostar is wonderful and aside from the market in the Old Town with the Stari Most bridge, there is a walk you can do through the city that was referred to us as the "war walk." It is, as much of Bosnia is, filled with bombed and bullet riddled buildings from the war. There is also a large cemetary where most of the deceased are not older than 23, all fighters and victims of the war. We stayed at a FANTASTIC b&b called the Shangri-La, right across the street from part of the Old Town. A couple owns it and they also live there. They are incredibly nice and the place is immaculate and lovely. Breakfast is great and you get eggs cooked how you like them, cereal, bread, and other nice things. I definitely recommend it!

In Sarajevo we stayed at the Halvat Guesthouse, which is located right inside the Old Town. Prices are about 66 euro a night right now for double occupancy with breakfast, which is a bit high but the place is quaint, well located and quiet. Here they also make you eggs for breakfast and nutella pancakes (delicious.)
In Sarajevo you must go the the Historical Museum. It chronicles the Yugoslavian Wars from 91'-'95 best and is very open and well put together.

For timing, you could see major parts of both Mostar and Sarajevo in 2 days each. We did Mostar in one day and had just under 3 days in Sarajevo and found we saw what we needed to see.

If you're thinking about any other places in Croatia, save yourself the trips to Zadar and Split- they're not really worth it. The towns and villages on the coast between the major cities have much more character, but Dubrovnik was lovely and worth a good few days. Enjoy your trip!

I can't help with driving to Sarajevo, we took the bus from Dubrovnik. We stayed at Hotel Michelle in Sarajevo. Fantastic hotel-just a very short walk to old town. The nicest staff, immaculate rooms, and breakfast to die for. In Dubrovnik, we stayed at Apartment Placa right smack in the middle of old town.