In 1834 construction began on the Baldwin Home and was completed in 1835. The thick walls were made of coral and stone. The structure was sturdy consisting of hand-hewn timbers. In 1840, a bedroom and study was added, and in 1849, an entire second story was completed.