I arrive in MAD from Milan @ 10:00 AM on Sunday 6/3 and my flight back to Miami departs MAD @ 3:00 PM. In your opinion, will I have enough time to go into the city center for brunch? Thank you for your help

Anything is possible, but Sunday Brunch in Madrid falls into a different realm, especially since you have to be back at the airport at least two hours before departure on international flights.

Is there a big distance between the airport and the city center?

The airport is about a 20-minute taxi ride from city center, depending on traffic.

The problem with Sunday brunch is not many places serve brunch, and lunch usually doesn't begin until 13:30 (1:30 pm).

Maybe you can find a place where they serve breakfast till 12

In my experience in the Madrid airport for an international flight you should be there a minimum of 2 hours prior. Sometimes check-in lines are very long. Are you flying the same airline through? If not , you might have to claim your lugagge and check it again. There might also be a change in terminal involved.
Even if that is not the case, I find a window of 5 hours not enought to do what you want, actually not enough even to go to Madrid just to walk around.

Most restaurants don't open until 1pm or 1:30pm but you can certainly find a bar to serve you sandwiches or tapas-like foods.

On a Sunday the traffic SHOULD be very light so you probably won't have trouble getting into/out-of the city quickly and easily by taxi.

After you get through customs and to the taxi stand you'll probably have a total of 2 hours time to get into the city, eat, and get back to the airport 2-hours before your next flight.

Personally, I wouldn't try it - too stressful - and you'll ONLY have enough time to eat a sandwich or food-snacks and no time to see anything, constantly distracted by the time.

Saludos, MadridMan
--
Madrid, Barcelona, & MORE @ you-know-where!

So I guess it will be shopping @ the airport. Thanks!!!