In 1852, Dr. Andrew Oliver built a large brick Italianate style house on Main Street in Penn Yan. This 18 room home was a wedding gift for his son, William, who also pursued a medical career. William had three children, Jennie, William (another doctor), and Carrie who lived their entire lives in the Oliver House. As none of these siblings had children, the family line ended with the death of Carrie in 1942. It was Carrie’s will that deeded the house to the village of Penn Yan.