In 1823, Congress purchased this 115-acre site, which would ultimately be used for two federal hospitals. Not until 1832, however, did it appropriate funds for construction of a hospital to serve the nearby Boston Naval Shipyard (CH15) in Charlestown. Alexander Parris was hired to design the hospital, and his earliest plan is dated January 26, 1831. Parris was instructed to design a building that could be built in segments, and his proposal consisted of a long rectangular center section with two end pavilions. The center section was to have a three-story veranda with iron balconies and Greek Revival detailing. When funding was provided initially for only the east pavilion, Parris prepared plans in September 1832 for a freestanding structure.
This building was constructed of Vermont granite in 1835–1836 and comprises the east three bays of the present granite structure. When the hospital was enlarged in 1865, the decision was made to simply extend the building to the west rather than following the architect's original scheme. A north wing was added to this extension in 1903.