I would move to a hotel near the airport. That way you won't be stressed if there is an accident or some other cause for a slow moving freeway. You would have to get up pretty early to drive to the airport 2 hrs before your flight, drop the car and check in...esp. with a big family in tow.....

Ditto Kleroux' comments. It only takes a 2-car fender-bender (even in the other direction lanes) to cause major delays at any hour.

You could even turn in the car the night before your flight. Many hotels in the airport area have shuttle services to places like Manhattan Beach (downtown and mall) where there are many moderately-priced restaurants.

Sunday morning shouldn't be bad. But being from out of town you probably wouldn't have alternate routes in mind should you find a traffic snarl. It would be much less hastle on the travel day to spend the night before near LAX.

There are a few hotels, like the Embassy Suites LAX and Marriott that are walking distance from a couple of the rental car companies (Budget, Avis). The Embassy Suites, and I think a Four Points, are across the street from those car lots. Worst case, you could drop the family off, drop the car, take the car shuttle to the airport and the hotel shuttle back to the hotel.