On 30 March 1791, Pierre Charles L'Enfant and George Washington walked over the area designated for the federal capital and decided on the general locale for the President's House and executive department offices. In a letter to Washington that June, L'Enfant discussed in detail the site and his design for the “President's Palace.” Located on a high ridge extending east-west, L'Enfant's final site selection was predicated on an advantageous southern vista down the Potomac and on proximity to the Capitol and legislative offices. L'Enfant described the President's House as a combination of the “sumptuousness of a palace” with the “convenience of a house and the agreableness of a country seat.” 33Although its plan has been partially obliterated on the only surviving manuscript map attributed to L'Enfant, the engraved plans (the first of which appeared in March 1792) show a complex, U-shaped arrangement of garden mound, forecourt, and flanking buildings leading to an elongated central block with a curved portico traversing the entire north facade. The two flanking buildings (shown with incomprehensible floor plans on engraved maps) were probably intended to house the executive department offices, then four in number.

In an April 1791 letter to L'Enfant, Thomas Jefferson noted his own preference for a President's House based on Parisian “modern” architecture, citing seventeenth- and eighteenth-century examples. There is no known response by L'Enfant. The competition for a design was announced in March 1792, just after L'Enfant was fired. Recognizing the limited resources of the government, Jefferson suggested that the competitors be told that the central section of the building might be constructed first (“with the appearance of a complete whole”), while the detached wing buildings could be erected later. Designs by six competitors survive, although nine entrants are known. Several sketches by Jefferson exist, but many probably date from his presidency between 1803 and 1811.

Between 1798 and 1800 two executive department office buildings, the Treasury Department and the War Office, were erected contiguous to the White House. Designed by George Hadfield, they were large, rectangular buildings set at right angles to and aligned with the White House on the east and west. Like the President's House, which they were meant to complement, they were derived from Anglo-Georgian prototypes of a half century earlier, with raised basements and water tables, central projecting Ionic porticoes, belt courses dividing two equal stories, and high, dormered roofs. Burned by the British in 1814, they were rebuilt by James Hoban, who erected two additional buildings, one for the State Department and another for the Navy Department. They nearly replicated Hadfield's two earlier buildings and were located to the south of the extant buildings on 15th and 17th streets, thus framing the executive mansion with the State and Treasury buildings on the east and the War and Navy department buildings on the west. The Treasury Building, occupying the southeast position in the group, was a victim of arson in 1833. Robert Mills's new Treasury Building was begun in 1836; the State Department Building to its north was not demolished until 1867, during the last phase of the Treasury Building's four-part construction, which took place over more than three decades. Both William Strickland and Mills provided designs to enlarge the buildings for the War and Navy departments in the 1840s, as did Thomas U. Walter in the 1850s. The present Old Executive Office Building was built as the State, War, and Navy Building between 1871 and 1888, following a design by Alfred B. Mullett. The Treasury Department continues to occupy a building in the presidential park, but the other departments are dispersed throughout the region.

Major architects worked on memorial sculpture and miscellaneous buildings designed for, or now located within, the White House grounds. Two gate houses (c. 1828), miniature cubes that were originally the west entrance to the Capitol grounds, were moved to the southern corners of the Ellipse in 1874. Charles Bulfinch copied their articulation of channeled rustication and guilloche frieze directly from William Thornton's design for the basement story of the Capitol. Simple Roman Doric porticoes are surmounted with a rectangular panel of sculpted ornament identical to decorative panels Bulfinch had used in the Capitol rotunda. The McMillan Plan had designated an area south of the State, War, and Navy Building as the location for a monument. Cass Gilbert's First Division Memorial ( WH01), an 80-foot, monolithic Doric column in pink Milford, Massachusetts, granite, was erected there in 1924. A private tribute by members of the First Division of the American Expeditionary Forces to commemorate their comrades lost in battle in France between 1917 and 1919, the slender column with its gilded, bronze-winged Victoryby Daniel Chester French is based on Joseph-Louis Duc's July Columnin Paris (1840). In choosing the delicate Parisian column (rather than the traditional Roman model with a helical sculptural frieze), Gilbert was doubtless considering its proximity to Mullett's French-inspired building, as well as responding to the taste of his own time. The names of all the dead from the 1830 Revolution in Paris had been inscribed on Duc's column, but Gilbert placed the 5,599 names of the First Division dead on bronze tablets set on the sloping surface of the main plinth, where they can be easily read. Two walls, to the east and west, which create an enclosure for the column, were designed by Gilbert's son in 1957; the west wall bears the names of the First Division dead in the Second World War, the east wall those who died in Vietnam (added in 1977).

The Second Division Memorial ( WH02) (1936), located near Constitution Avenue just off the north-south axis of the White House, is John Russell Pope's modern rendition of a triumphal gateway. A single portal, framing James Earle Fraser's 18-foot gilded bronze sword, was cut from pink and gray Connecticut granites. The inside edges of the square-arched opening are curved, giving relief to its relentlessly rectilinear form. Two L-shaped arms were added in 1962 to memorialize the Second Division's losses in the Second World War and Korea.