So many young men wanted to come to “General Lee's College,” as it was often called after the Civil War, that they could not fit into the small room on the colonnade that the college had been using for a chapel. President Lee asked the Board of Trustees in 1866 to consider building a separate building “devoted exclusively to religious worship and instruction.” The chapel was designed by a VMI engineering professor, who also taught the first architecture course at the school and even wrote his own text book for it. He seems to have been influenced in his chapel design, at least in part, by James Renwick's Romanesque Smithsonian Institution (1847) in Washington, D.C. The picturesque character of the flaring tower, the brick and wooden drip moldings, and the narrow, round-arched, cross-mullioned windows all offer a startling contrast to the classical colonnade that the building faces.
Robert E. Lee died in 1870, and the Lee Memorial Association was formed to create a suitable monument. They commissioned Richmond sculptor Edward V. Valentine to carve the recumbent statue of Lee asleep on his camp bed. It was not until 1883, however, that the college raised the money to complete the mausoleum addition to the rear of the chapel. In 1962, with the help of a Ford Motor Company grant, and the professional guidance of a Lynchburg firm, the university did extensive work to preserve and modernize the building. The lower basement area serves as a museum, and was further renovated in 1999.