Im not interested in going to Pushkar for the camel fair but is it worth visiting at other times?

IME no. I would not return. Several bad experiences in one day:
1. As I had my wallet out to pay the cab driver I sensed a man walking towards me, caught his eye, saw his cunning look, stared back at him, gripped my wallet tightly and he veered away.
2. A so called student offered to show us inside a temple, said he would do it for free and then demand money after we had finished.
3. So called priests wanted money just to allow me to take a picture by the lake and yelled when I took a pic and did not pay them.

Just be careful and forewarned if you go!