By 1864, Mary Greeley was ready to move. For the past 10 years, the Greeleys had spent the summer months in a house deep in the woods of their 78-acre farm in Chappaqua. They had found it increasingly dark, dank, and remote. Furthermore, after the Draft Riots of 1863, when an angry mob had nearly burned down the office of the New York Tribune, they became concerned that their isolated home might be unsafe. So, in the spring of 1864, they seized the opportunity to buy a small house on Main Street (now King Street), right at the farm entrance. They enlarged it, rearranged the rooms inside, and moved in before the end of the year.

Horace and Mary Greeley continued to make it their summer home until their deaths in 1872. The following year, their daughters Ida and Gabrielle spent one last summer there, together with their aunt and her two daughters. One of these cousins kept a diary that was later published, and that contained a detailed description of the house. In recent years, this document proved invaluable in the restoration of the building.