The barracks are located just north of the site of Fort New Richmond, the dirt fort the British built in 1779 during the American Revolutionary War, which was quickly captured by Spanish forces in September of that year and renamed Fort San Carlos. In September 1810, West Florida revolutionaries seized the fort from the Spanish and flew the flag of their republic over it, but they surrendered peacefully to the United States in December 1810. As the westernmost fort in the United States at that time, it theoretically provided protection against the Spanish from the direction of the Sabine River, as well as from slave rebellions. (A major insurrection had taken place in 1811 along the river parishes.) The dirt fort was obviously inadequate, and a substantive expansion was initiated under the direction of army engineer Gadsden, who later negotiated the Gadsden Purchase of land from Mexico in 1853.
The four two-story buildings enclose a pentagonal courtyard that is now open on the river side. Originally this fifth side accommodated a commissary-quartermaster structure and adjacent ordnance warehouse, but these were of shoddy construction and subsequently demolished by the 1830s. Monumental Tuscan columns and two-story galleries were built across the fronts of the four buildings facing the courtyard, and identical columns and galleries were added in 1834 to the outer facades, where windows had been installed in 1826. All four buildings are covered by hipped roofs.
Lieutenant-Colonel Zachary Taylor, who had been posted to the barracks more than once, was in residence there when elected president in 1848. During the Civil War, both Union and Confederate armies occupied the barracks. Records indicate that the barracks have houses many notable figures, who either were garrisoned there or stayed while passing through Baton Rouge; the list includes William Tecumseh Sherman, Jefferson Davis, Ulysses S. Grant, and Abraham Lincoln. The Baton Rouge post was deactivated in 1879, then leased by Louisiana State University (LSU) and used as dormitories until the university moved to its new campus in 1925. The barracks then accommodated the offices of various state agencies. Today the buildings house a museum, the lieutenant governor's office, and apartments for legislators.