The Rogers Mansion was built by Captain Albert Rogers in 1843, at the peak of the whaling industry. The property had been in his family since 1648. In 1898 Samuel L. Parrish, a wealthy NYC attorney and land developer purchased the home and added many colonial-revival additions. The Museum acquired the one-acre property in 1952 and began adding historic 19th century trade shops and a barn, saving them from destruction.

Today the Rogers Mansion Museum Complex has 12 historic buildings listed on the National Registrar of Historic Places. The Rogers Mansion is open Feb-Dec. The Museum also manages three other historic properties. The Pelletreau Silver Shop at 80 Main Street was built in 1686 and is the oldest continuously opened trade shop in the Americas. It is open year-round. The Thomas Halsey Homestead at 179 South Main Street has a 1666 first period house open to visitors during the summer. Conscience Point Historic Marker and Nature Walk is located on North Sea Road in North Sea, NY. The five acre wildlife preserve is open sunrise to sunset and has a shellfish hatchery with education programs during the summer.