The oldest part of the house, two rooms up and two rooms down, was built for Reverend Hale in 1694. In 1745, Colonel Robert Hale, Jr. (John Hale's grandson), added the front section of the house with the gambrel roof. He excavated a cellar (where evidence of a buttery has been found), added the present staircases, and built an addition of two rooms down and two up, along with the gambrel roof that allowed rooms on the third floor. The property at that time included about 100 acres and extended to the ocean.