Baker Mansion was originally home to ironmaster Elias Baker and his family. Baker purchased the nearby Alleghany Furnace in 1836 in partnership with his cousin Roland Diller. Elias brought his wife, Hetty, and their two sons, David Woods and Sylvester, from Lancaster County to what was described as a “tolerable good mansion house” near the furnace.