My family and I will have a four hour layover in Athens. It will be late morning through early afternoon on a Saturday in August. Is it crazy to even try to get out of the airport with such a short amount of time?

I think so. There's a Sofitel or something of that sort in the terminal. I'd suggest having a long brunch there. Don't forget that you have to be back to the airport at least an hour or so ahead for a domestic flight; much longer for international.

4Hrs is not long enough to go anywhere once you allow for disembarking, customs & check in for next flight.
There is plenty to keep you occupied, apart from shopping! There is a small museum at the airport, The Olive Tree restaurant, lots of smaller cafes & fast food. As Weales says, the Sofitel is an option too, situated right across from the terminal.
Take a look at this airport site for a fuller list of eating, shopping & all other facilities available,
http://www.aia.gr/pages.asp?pageid=977&langid=2

The Athens airport is a long hike to the city. You might get away with it in Amsterdam, but not Athens...

Thanks everyone. I thought it might be a long shot. Sounds like we will be visiting the museum and having lunch either in the airport or at the hotel next door. Might be wise to find someplace to take a nap if that's even possible in the airport in hopes to be a little fresher for our final destination.