About 1730, Yorktown merchant Thomas Nelson erected the present dwelling on an urban estate that extended down to the water's edge. The house remains largely intact, despite serious damage sustained during the Revolution and the Civil War. With its two-story bulk and double-pile footprint, the Nelson House was an extraordinary creation for its time and place—one of the earliest realizations of the so-called Georgian plan in this region. In outward form and internal arrangement, it resembled Berkeley, seat of the Harrisons in Charles City County, with which it was nearly contemporary.
Thomas Nelson appears to have imported brick for his new house, as the chocolate-colored bricks with their yellow flashing resemble nothing made in the Virginia colony at the time. Together with the Portland stone elements and shiny rubbed bricks, they resemble materials employed in southern England and probably came from that region.
Owing to its early date, Nelson's house exhibited an unusual degree of ornamental vigor, inside and out. On the exterior, sandstone quoins define the corners while openings are picked out in bright red brick with Portland stone accents. The rubbed brick doorway is a characteristic and early example of the form. The full effect of the embellishments was reserved for the two street elevations; the plainer work, which now faces the garden, originally fronted on the kitchen or service yard. Inside, turnings and architectural orders in the public rooms were executed with license and imagination.
Early in the 1920s, George P. Blow of La Salle, Illinois, renovated the dwelling, assisted by New York architect Percy Griffin and Richmond landscape architect Charles Gillette. Griffin made extensive changes, adding dormers and areaways to maximize usable space in the attic and cellar. Early in the 1980s, York Hall, as it was then called, was acquired by the National Park Service and subsequently restored to its original appearance under the supervision of NPS architects Lee Nelson and Robert Simmonds.