Samuel Young built Gloucester, a two-story, five-bay brick house, but after the death of his wife, Anna Frances, in 1804, he sold it in 1807 to Winthrop Sargent, former governor of the Mississippi Territory. Sargent married a wealthy widow, Mary McIntosh Williams, and immediately enlarged the house, excavated the dry moat in front, and renamed it Gloucester. He created an eight-bay north facade with two matching entrance doors, each with paneled pilasters, an entablature, and a fanlight, and echoed on the east the existing semi-octagonal west end. Sargent’s Gloucester had a single-pile plan with small cabinet rooms flanking a double-tiered rear loggia. Two stair halls separate each of three large rooms. New owners James and Catherine Wilkins added a giant-order portico and a rear colonnade probably before 1830, and the middle of the three rooms was decreased in size to create a front cross hall elaborated with repeating arched openings. Two original two-story outbuildings include a brick kitchen with a gabled roof and a brick billiard hall with hipped roof, both with slave quarters above.