James Pinchot acquired one of the last natural sites around Milford and in 1884 commissioned his friend Richard Morris Hunt to design a summerhouse that was to be based on a source dear to the Franco-American community, the Marquis de Lafayette's LaGrange near Paris. Pinchot knew Hunt from their shared work building the Statue of Liberty in the New York City harbor, a project that occupied Pinchot after his retirement in 1875. How much of Grey Towers is LaGrange and how much is Hunt's contemporary interest in large conical forms is unclear but the net result is one of the great buildings of Pennsylvania. Sited to provide views of the town, the great bend of the Delaware, and High Point in New Jersey, it is Victorian in its dominance of its setting. Hunt chose the regional bluestone to accent rubble walls built of local stone and
What had been a large house on a bald hillside in the French manner during the era of James Pinchot was transformed after James's son Gifford (1865–1946) married Cordelia Bryce in 1914. Gifford had learned to love the local forests and streams in his trips to Grey Towers during his Yale years and to those can be attributed his lifelong interest in the environment. After study abroad in the French school of forestry at Nancy, he returned to the United States to manage George Vanderbilt's Biltmore estate in North Carolina. In 1905, fellow Republican Theodore Roosevelt appointed Pinchot as first head of the U.S. Forest Service, and with Roosevelt's connivance established vast tracts of western lands as National Forests. In 1922 and again in 1930, Pinchot served as governor of Pennsylvania, running a patrician campaign from his summerhouse, Grey Towers.
If Gifford was at home in the forests, Cordelia, raised on Long Island, had an interest in architecture. She sought to transform both house and landscape from her father-in-law's vacation home to a setting that would support their vigorous and intense life. She began on the interior by forming larger rooms out of the cluster of tiny spaces that she had inherited. Gifford recalled for the Saturday Evening Post in 1922 that “the architects said she couldn't do it, for it involved removing partitions, beams and a central fireplace. But she did. And, of course, it's a vast improvement. She's revised and edited and altered this house practically beyond recognition. She will never be done. As soon as one thing is finished, and I think we'll have peace, she breaks out in a fresh spot, and the hammering and revising begin anew.” After Cordelia had the house in shape, she decided to bring the delights of the adjacent landscape into proximity with the house. In 1923 she designed a rustic pergolashaded swimming pool to re-create the pleasures of Sawkill Creek where Gifford had learned to trout fish. That feature was then connected to the house by an outdoor dining terrace complete with the remarkable “Finger Bowl” table, an irregularly shaped water-filled concrete bowl that could seat two dozen around its Baroque edge. Places could be set on the broad lip and food was passed on floating bowls, to the delight of all. Small buildings were added in consultation with Chester Aldrich of New York City's Delano and Aldrich. The “Bait Box,” a playhouse for their son Gifford Bryce was begun in 1923, while the “Letter Box” (1925) served as Gifford's office and archive. The raised terrace and moat were the next projects of Cordelia and Chester, necessitating a revision of the drive from the usual serpentine Victorian approach to a more stately passage through a grand allée of trees. The last feature of consequence is a brilliant carpetlike mosaic of stones and hot-pink cement in striking wave motifs that recalls the Pinchots’ trip to the South Seas and was crowned by the splendid oval pergola above the Finger Bowl. These were the result of Cordelia's collaboration with New York City artist and architect William Lawrence Bottomley between 1932 and 1933.
The Pinchot family donated Grey Towers and the estate to the U.S. Forest Service in 1963. Dedicated by President John F. Kennedy in September 1963, it suffered the usual indignities of federal utilitarian care, from “eye-eze” green walls to the institutional green tiles of the toilets. Under the inspired direction of site director Edgar Brannon and landscape architects Andropogon Architects, the house and landscape have been restored to the Cordelia and Gifford Pinchot era and now serve both the U.S. Forest Service Leadership Program and the Pinchot Institute for Conservation. It is the most remarkable site in upstate Pennsylvania and an absolute must-see.