Waterloo is one of five pre-Revolutionary War houses that represent the height of architectural achievement on the Eastern Shore. According to architectural historian Paul Touart, few Somerset County planters of the mid-eighteenth century aspired to such ostentatious architectural statements as Waterloo’s owner, Henry Waggaman, one of the county’s wealthiest planters and a member of the Provincial Assembly of Maryland. Like others in the region, Waggaman likely grew a combination of tobacco, corn, and wheat utilizing enslaved labor. Waterloo exhibits exceptional brickwork, including glazed checkerboard and diaper patterns, highlighted by contrasting stone quoining. Rather than adopt the traditional center-passage plan, Waggaman chose a four-room plan manifested in a two-and-a-half-story, pedimented gable-front main block, the first of its kind in the area. A stair hall with an unusual three-flight twisted-baluster stair sits to the front of a large parlor, with adjoining rooms with corner fireplaces to the other side. A hyphen and kitchen wing were added later.
Waterloo was owned by several prominent local families during the nineteenth century before it was purchased by the county for use as an almshouse, from 1864 to 1948. Once again a private residence, the flanking hyphen and bedroom wing were added in 1959. None of the plantation-era outbuildings remain, although the office of physician Littleton Handy, who owned Waterloo briefly during the 1850s, is extant. It may have been Handy who first called the property Waterloo.