St. Mary's is the gift of Sarah Gibbs, who not only made the gift of the church, but also bequeathed to it her eighty-acre farm, Oakland (the name later adopted for the principal Vanderbilt farm, which once surrounded the church and its graveyard). Previous to providing for St. Mary's, she had already called on Richard Upjohn to design another Episcopal church in Middletown, Holy Cross (see entry, below). Whereas Holy Cross is shingled wood, St. Mary's, of local field-stone, takes its cue and its appeal from the tradition of English country churches. It reflects Upjohn's active interest as an architect and pious Episcopalian in the reform movement promoted by the English Ecclesiological Society, which called for a return to medieval tradition in church liturgy, building, and fittings to counter eighteenth- and early-nineteenth-century tendencies toward the simplification of High Church ceremony and ambience. Specifically, St. Mary's derives from St. James the Less in Philadelphia (1846–1848), a key church in the ecclesiological movement in America. Behind this, it seems to be closely related to St. Michael's, Longstanton, in Cambridgeshire. In accord with ecclesiological principles, St. Mary's makes an explicit statement of chancel, porch, and sacristy as visible projections from the main box of the church.
Entrance to the church is through the side porch rather than the front portal. The plaster interior, with wooden gable rafters and wall brackets forming simple roof supports, is appropriate if unexceptional. Sarah Gibbs gave the memorial tablet opposite the entrance door in memory of her parents. In the marble relief, they appear in ancient Roman garb. A nude angel of death leads the husband and father away (he predeceased his wife by twenty years), while she mourns at an urn. It is the work of Horatio Greenough, commissioned by Sarah Gibbs when she was in Florence and completed in 1843 (before the building of the church), so the work of an eminent neoclassicist here joins the work of an eminent medievalist. Sarah Gibbs herself is buried immediately behind the chancel of the church she sponsored, under a casketlike memorial from yet a third major American designer, Richard Morris Hunt.