These moments are THE BEST but as you've noted, you cant plan them, just enjoy them as they happen. And it's not always 'away from the tourist sites' that you find a gem, it can be anywhere.

On my first trip to Rome, I was trying to find the colloseum one early evening, which the guidebook said was minutes from my hotel. I was walking round in circles and found myself on a wonderful street; I passed a grocer standing out on his step watching the world go by, there was an old fashioned pen shop, a bakery. The street opened out into a Piazza and I sat on the cool steps of a mossy little fountain fora few minutes. Feeling refreshed, and like I was in a Fellini film, I set out again and all of a sudden, there it was, the colloseum. The street was via dei serpenti and I've been back since but it's never been as magical as that first time.

it's not always cute streets, either. I will always remember stumbling into the middle of a Madonna of the Sea parade in a southern Italian harbour town, wandering deserted Chantilly one night and peering through tavern windows at the revellers and joining in a picnic (cheese, neat vodka) in the Romanian countryside. Don't try and recreate or seek it out, just enjoy it when it happens 😊

some of the interesting places that we have "found" in a city have become tourist "must dos" but that doesn't mean that they are no longer fun and interesting to go to, e.g. Rue Mouffetard in Paris.

I have often boarded a bus in a strange (or not-so-strange) city, ridden it to its end point, and walked back, finding all manner of discoveries along the way. Or sometimes, not.

It also depends on what time of day you're doing your wandering. In southern Europe, neighborhoods and villages are dead in the afternoon. Street markets often open early in the morning, closing about 1:00. Shops are open in the morning, closed for the afternoon, then reopen about 5 o'clock when workers shop for dinner on the way home.

I'd second Mimar's note on time of day. That was definitely a huge factor in Spain until we got properly onto local time.

One thing I've had some success with when in larger cities is breaking them up into neighbourhoods and researching the history and trends of each. I'll use google translate to search for cafe reviews in the local language and will try and find local online reddit-type forums where people are making jokes about what sorts of people you'll find where (you know, "here are where the old rich people live, and over there are the new immigrants and over there are all the university students, and this the tourist and fashion area, and that part is where the artists used to live but now it's mostly yuppie young families. Now the artists are over there!") and from that you can start to get a feel for what kinds of street character you might run into - tasty little start up restaurants or fashionable cafes or locals on a budget or tourists or wealthy elite.

Of course in many places what's 'touristy' is so for a reason and there's no hard separation between locals and visitors. They all like the same popular place to be. In other places they're a bit more separate, and if English isn't the main language then that google translating starts to come in handy. And of course sometimes you just get lucky! (Or think you've found a 'hidden' gem only to realize it's popular and well known after you return. )