The first William and Mary building, still standing front and center after three fires (1705, 1859, and 1862), was begun in 1695, four years before establishment of the town. It faced down the ridge between the James and York river drainages and so became the western terminus of the town's main street. Hugh Jones wrote in The Present State of Virginia (1724) that the building was “first modeled by Sir Christopher Wren, adapted to the Nature of the Country by the Gentlemen there,” creating a longrunning debate about the origins of the design with little prospect of resolution. The initial incarnation was a full three and one-half stories above a relatively high cellar and thirteen bays long, with a sizable hall in a rear ell—a remarkable edifice almost certainly unequaled by any other seventeenth-century building in the still rough-and-tumble Chesapeake.
After the 1705 fire, the facade was reworked with only two and one-half stories and a cupola that was apparently lower than the original. But the rear, shown on the famous copperplate probably engraved for William Byrd's now lost history of Virginia, retained three full masonry stories, including a loggia. There is evidence that the building was intended to grow into a full quadrangle, and Lord Dunmore apparently asked Jefferson to produce a plan in 1771 or 1772. Construction began but soon ceased. Yet by 1729–1732, when the chapel had been added as a second rear wing, its end wall was expensively finished with good brickwork and round windows, and the corresponding wall of the hall was rebuilt in the same manner, suggesting little interest in further enclosure. Likewise, when the Brafferton building was
After the 1859 fire, architects Henry Exall and Eban Faxon redesigned the building in Italianate garb, adding a pair of three-story towers at the front and enclosing the loggia to house classrooms and a stair. Remarkably, the walls survived another fire sufficiently to be used in a more sober 1867–1869 rebuilding with a wider center pavilion, sans towers. Following considerable debate, this pavilion was removed and the surviving seventeenth- and eighteenth-century brickwork was incorporated into a restoration that returned the building to its second appearance, including the chapel. Much of the 1928–1931 Wren-ish chapel interior is by Thomas Tileston Waterman.