On a trip from SF to LA, is there a big difference in the actual speed I can drive on hwy 101 vs I-5 ? Is the time it takes to make the trip really that much greater on 101 to make enduring the trip on I-5 worth it ?

I-5 set your cruise control for around 75 and stay in the right lane so all of the big rigs can zoom by. If you are looking for a boring road, this is it. Overall, 101 doesn't really take that much longer and is much more interesting (which again isn't saying a whole lot).

Curious

Your speed on 5 will average higher. The 101 is has cross traffic in many places with stoplights.
However, the 5 has got to be one of the most boring drives in the country. The 101 has some of the best California scenery of any roadway.

My experience is that there is about a 3 hour differential between taking 101 and I5.

My experience San Jose to LA is that Hwy 101 is about an hour longer than I5 - but it does depend on where in LA you are going. My experience is between SJ and LAX, which is on the west side of LA. If you are going someplace more central or on the east side of LA, then I can see how taking 101 would add even more time because of the drive across LA.

Google Maps predicts that taking I-5 (once you get to it) from SF to LA should take 5:49 without traffic but up to 7:10 with traffic. It predicts taking 101 all the way should take 7:04 without traffic and up to 8:20 with traffic.

I-5 is faster than Google predicts because hardly anyone goes the speed limit on 5. People take 5 to get to between LA and the Bay Area in the minimum amount of time. FYI my average speed on 5 the last time I did the drive hovered around 85, and there were plenty of cars passing me. In my youth, we would drive it late at night/early morning when it was even faster (and cooler).

Speed depends on when you are driving I5 though - last time we drove it, we were coming back to SJ on a Sunday and it was maddeningly slow. I think we averaged 50 mph and had to keep braking because the traffic would speed up, then a bunch of people would change lanes, and people would have to brake.

It very much has to do with your start/end points and time of day/day of week.

On the SF end, the diagonal trip out to I-5 can be a crunch in traffic, whereas if your start point allows easy access to 19th Ave/I-280/SR 85, the time required to be on "open" 101 can be less as you can skim past the Bayshore/Silicon traffic.

Once on the 101 it's hard getting much over 70-75 MPH through the Salinas Valley, owing to (a) cross traffic and (b) cops. OTOH once you're on I-5 past the I-580 junction it's hard not to go 80+ MPH without getting run over. Actually it's good NASCAR training - you learn how to slipstream people, and how to be slipstreamed (and that can be a bit nerve-wracking when all you can see through your mirror is the word "Peterbilt" spelled backwards.)

Things can go pear-shaped at the other end using either route. The Grapevine and associated terrain can be slow going as trucks, pickups and RVs lumber up the grade, and the Tejon Pass area is notorious for all sorts of bad weather - fires, windstorms, snow in winter, take your pick.

Then your reward is transiting the San Fernando Valley and downtown LA - IF that happens to be your route. If your route is to west LA, Beverly Hills or Santa Monica (or worse, LAX) then getting over to the west side poses problems of its own.

On 101, there are some slow patches past Santa Barbara where the old road is too small for the traffic, but there's enough variety in the scenery that one doesn't notice it. The real crunch comes in the aforementioned San Fernando Valley, where 101 can and does become a parking lot in the vicinity of the Great Enchilada Parking Lot, I-405. Again, if your destination is the west side, it's about your only choice, so enjoy your stay.

On average I'd say that from the Golden Gate Bridge to LAX airport using 101 usually "costs" around 2 1/2 hours compared to I-5. If you're starting in, say, Oakland and finishing in Glendale (you poor dear) the savings are closer to 3 1/2 hours.

2 1/2 - 3 1/2 hours is a small price for your sanity however.

What type of driver are you? Fast & risky, then take 5 & hook up with at least one other car going 80-90mph. Slow & steady, take 101 & drive with the grannies.

I often drive up the 101 with a grannie. She's my wife.