Here in 1774 Massachusetts's citizens signed the Suffolk Resolves, a resolution opposing the taxation policies of the British government. Beyond its moment of fame, the house represents three major periods of construction history at three different locations. Daniel Vose built the first section (right half) in 1764 at a site near the Neponset River. In about 1785, Dr. Amos Holbrook moved it down Adams Street, doubled its size, and added the one-story portico and the central hallway with its staircase and wood paneling.
Converted to commercial purposes by 1889, the house deteriorated so radically by 1950 that demolition seemed inevitable. Dr. and Mrs. James B. Ayer had it moved to their Canton Avenue estate and restored between 1950 and 1954 by William Morris Hunt, grandson of the artist and son of architect Paul Hunt. This work also included restoration of the aboveground basement “counting room” to represent the period when the Suffolk Resolves document was signed, even though this space was actually added in 1846. The house was deeded to the Milton Historical Society in 1963.