You can take a rental car from Croatia to Montenegro. I have done it on several occasions. Most likely the agency that told you that does not offer it because the insurance is higher. Just find a rental company that has the insurance and it won't be a problem. A good local company is www.eurocar-rental.com.hr/.

If you decide to take a bus instead, it is possible. However, buses leave Dubrovnik for Montenegro infrequently (maybe only once per day). Bus service within Montenegro is better and more frequent.

Agreed that you should be able to find a car rental agency that will allow you to travel to Montenegro. We used https://www.economycarrentals.com/Default.aspx and were very happy.

Tracy

This is the schedule for the bus station in Dubrovnik:
www.libertasdubrovnik.com/voznired.pdf
There are buses to Herceg Novi, Kotor and Budva every day.

In Montenegro, buses from Herceg Novi and Kotor are very frequent. This is the schedule for Kotor bus station:
www.autoboka.com/red_voznje.html

Thanks, everyone! However, none of these car rental places seem to allow dropping off the car in Podgorica, Montenegro. We are not returning to Dubrovnik, but wanted to drop off the car in Podgorica before catching a flight out from there. Anyone know if this is possible? Thank you for all the recommendations!

My impression is that renting in one country and dropping off in another is often difficult or not allowed.

In this case part of the problem could be that, as my landlord in Dubrovnik reported, "Montenegro is the stolen car capital of Europe."

How about taking a bus from Dubrovnik to Kotor, then renting the car there? I made that drive two years ago and it took only about 1 hour. Alternatively, I believe there are boats from Dubrovnik to Kotor, although I don't have schedule information in front of me.