Monroe County's most impressive antebellum mansion overlooks the town and its outskirts, as well as the Walnut Grove estate, from its hillside site. With a monumental pedimented portico fronting the central three bays of its five-bay brick facade, Elmwood appears at first to be an architectural transplant from the deep South, perhaps from Natchez. Actually, its architectural pedigree descends from a closer source. In all likelihood William B. Phillips (c. 1790–1861), one of Thomas Jefferson's most trusted and accomplished brickmasons at the University of Virginia, designed and built Elmwood. After completing their work for Jefferson, Phillips and others employed at the university received contracts throughout Virginia, gradually evolving from builders who followed their mentor's plans to architect-builders working from their own designs. Phillips was listed as a taxpayer in Monroe County in 1836, and several years later he and Elmwood's owner were involved in a suit relating to money owed to Phillips for work at Sweet Springs.
The portico at Elmwood is of the Doric order as found at the Theater of Marcellus in Rome, the same order used at Pavilion X at the University of Virginia, which Phillips and John Neilson built. It is also the same order used at the nearby Sweet Springs Hotel (see entry below), which Phillips is credited with designing and building. Both Elmwood and the hotel are raised above full English basements; both have recessed, plastered panels between the windows of the two main floors; and both have lunettes centered in the pediments of the porticoes. Both originally had Chinese Chippendale railings on the porticoes, but only those at Elmwood remain. These railings are the likely progenitors of numerous other examples of this pattern found throughout the springs region. The portico pediment shows an extremely unorthodox placement of triglyphs along the raking angle.
Elmwood has a double-pile, center-hall plan.
As might be expected, Elmwood's owner cut quite a figure in his day. Hugh Caperton II was a wealthy merchant and member of the state legislature and served in Congress before building his mansion. Elmwood was originally surrounded by many service buildings, including an earlier family house that survived until c. 1906. The extensive collection of farm buildings that now stand to the rear dates from the early twentieth century.