The original house, of which the present structure is a rebuilding, was erected in 1836 by Matthew Saint Clair Clarke, former clerk of the House of Representatives. A five-bay, hip-roof cube, it was devoid of surface decoration other than two central tripartite windows and an elegantly simple marble Ionic porch, never installed, which is now on the Enoch Pratt House in Baltimore.
In 1854, Thomas U. Walter transformed the house into an Italianate palace for a new owner. The exterior was stuccoed and was scored to resemble ashlar, and the elaborate sandstone window and door frames were added. The proportions were also altered by raising the roof and adding a cast-iron bracketed cornice; the present mansard roof was added in an 1877 renovation, as was the front door. The ground-floor windows (raised above high basement windows screened by cast-iron panels) are elaborately treated with substantial balus-traded balconies carried on consoles. The balconies and consoles are of sandstone, but the balusters are of cast iron. The window frames with boldly projecting triangular pediments supported by brackets are similar in both proportions and details to Walter's window treatment on the principal story of the Capitol extension. Bracketed flat lintel surrounds are used on the second story while the third-story windows are simply framed. The entrance lamps date from Walter's time.