Only the first story remains of a much larger Romanesque Revival building, which lost its upper stories and steep hipped roof in a 1956 fire. Named for Robert Rogers, president of the Eagle Bank, the building originally mixed philanthropy and finance, combining a public library above with ground-story banking rooms and the local YMCA. The architect was Stephen C. Earle, of Worcester, Massachusetts, one of the first graduates of MIT's department of architecture. Like much of his work, it is Victorian Gothic in spirit, strongly influenced by H. H. Richardson's hefty Romanesque masonry.
Wallis E. Howe reconfigured the damaged library, which, in its truncated form, provides an interesting juxtaposition to the adjacent U.S. Post Office (1961–1962, Philemon E. Sturges; 515 Hope Street). Sturges's building is itself a hybrid, incorporating doors and windows of the Wardwell House (1815, Russell Warren), which once stood on the site.