As you're in a B&B, you'll probably be off fairly soon after breakfast so your plan is certainly do-able. It does depend on how long you spend at each stop.

You'll need a couple of hours for the Caves and then lunch and hiking depends on how long you want to spend on them.

It certainly is do-able - your biggest issue though is going to be getting from wherever you're starting out from to Castleton. If you're leaving nice and early - say half seven or eight, after a wonderful b&b cooked breakfast (I'm jealous, can you tell?) you do stand the distinct possibility of picking up some rush hour traffic around Leeds/Bradford and Huddersfield/Halifax - depending on your route. Don't asume that sticking to the motorways will avoid the issue - they'll be busy with commuters too. However once through that you should be able to make good time to Castleton abd be there by mid-morning allowing an hour or so for one of the caverns and maybe a visit to Peveril Castle (you can start reading Walter Scott now!) before driving to Bakewell for lunch and an afternoon hike at Ilam.
Just watch what time you get back in the saddle again - your route to Rutland will likely take you to Nottingham and thence on the A606 via Melton Mowbray to Oakham. Once on the A606 you should be fine but getting round Nottingham anytime between 4:30pm and 6.30pm will be a major frustration. Your main issue is going to be the need to cross over to the south of The Trent, for which, for your purposes there are two options - the ring road and Clifton Bridge - which I'd probably recommend for you but it will be slow going along the length of that portion of the ring road you'll need to take or, more directly, through the city and over Trent Bridge to West Bridgford and the A606 which is to all intents and purposes a complete mare of a route for those relying on a GPS and not able to avail themselves of the various shimies that the locals rely on.
There is an alternative and that's to come off the M1 at J24 (or travel south from the Peak district through Ashbourne and pick up the A50 eastbound which meets the m1 at the same junction) and take the A453 towards Nottingham and briefly take the A52 ring road south and thence the A606 but beware, the A453 itself is an attrociously over subsribed route into the South of Nottingham and generally has lengthy delays at all times of the day.

That all sounds doom and gloom buit it isn't - all I'm saying is prepare yourself for delays around Leeds in the morning and Nottingham in the evening.

And another thought - depending on what time you are travelling in the afternoon, early evening you could do a lot worse than take a right turn off the A606 in Nether Broughton, signposted Old Dalby and find the Crown in Old Dalby for dinner. Some of the best cooking in the East Mids, with a fantastic reputation which is very well deserved. That could put you at your B&B, fed and watered after a long day, for 9pm ready for an early night.

Have a great trip!

Dr D.

bookmarking