If you're familiar with the Hamptons’ upscale vibe, you'll soon realize that, though it's in the Hamptons, Sag Harbor is not quite as "Hampton-y" as its other destinations. On the South Fork's north coast, this historic hamlet has a strong maritime flavor that stems from its history as a whaling port. The first white settlers arrived in the late 1600s, learned a thing or two about whaling from the resident Native Americans, and started sending out whaleboats in the mid-1700s. By the time the industry hit its peak in the mid-1800s, Sag Harbor had become one of the world's busiest ports.
The centuries-old Main Street, lined with boutiques, galleries, and restaurants, leads to the wharf, which for centuries has served as the heart of the community. Once, tall ships from around the world would drop anchor here, but today it is impressive sailboats and powerboats that line the marina and bay, often including the mega-yacht of the Mayor of Margaritaville himself, Jimmy Buffet. Thanks to careful preservation, much of Sag Harbor's 18th- and 19th-century architecture remains intact, including Greek Revival houses once owned by whaling captains, early-colonists' homes, and Victorian houses built for wealthy industrialists.
Start your visit down by the harbor at the windmill that houses the visitor center. The enthusiasm for their much-loved town of the volunteers here is truly infectious.