I'm going to be in Serbia and plan on visiting Istria next. Due to the lack of direct flights I was thinking of flying to Ljubljana and renting a car to travel through Istria, returning the car in Zagreb, and flying out of Zagreb. Has anyone done something similar? I was wondering if there are any issues with traveling across borders with a rental car?

You will need to make sure that the rental car has a vignette that is required in Slovenia. And you will probably pay a hefty fee for renting in Ljubljana and dropping off in Croatia. There are frequent rapid trains (and probably buses) from Ljubljana to Zagreb and probably some other options, so I would recommend separate rentals: Use a car rented in Ljubljana to visit a bit of Slovenia and then return the car In Slovenia. Take public transportation to Croatia. Rent another car for your time in Istria. Return the car in Croatia.

I suppose you absolutely want to visit Ljubljana. Otherwise this detour wouldn't make any sense.
You may take one of the 6 daily buses from Beograd to Zagreb (journey time 6 hrs) where you will find frequent buses to all destinations in Istria.

Take the bus to Zagreb and hire a car. Go to Istria or Ljubljana, no problem to cross the borders with the hire car.

I second Kevina's advice. If taking the bus doesn't suit you, you could also take the train to Zagreb. Flying into Ljubljana only makes sense if you'll return the rental back to Slovenia and then either take the train from Ljubljana to Zagreb or fly out of Ljubljana. One-way rentals are extremely expensive.

Yes, you can take the train too, but that will take the whole day.

I know I’m in minority, but I like your plan, both Croatia and Slovenia are wonderful. A car will give you more flexibility and will save some time.

>> You will need to make sure that the rental car has a vignette that is required in Slovenia.
If the car does not have one, you can buy one at the border.

>> And you will probably pay a hefty fee for renting in Ljubljana and dropping off in Croatia.
We almost rented in Slovenia and returned the car in Croatia last year (we ended up not doing it because we were able to change our flight back). Yes, there would be a fee for dropping off in Croatia, but that will minimize your back-tracking and will give you more flexibility.

You don’t say how many days you have, but consider Plitvice Lakes in Croatia, and as an alternative, returning the car back in Slovenia. Zagreb is only a couple hours away from Ljubljana.
Have a great trip.