As built, Holly Grove was a two-story, doublepile brick house with a rear ell, larger than most of its contemporaries. David Ruffner, its builder and owner, was a leading salt manufacturer in the Kanawha valley. After the James River and Kanawha Turnpike was built in front of his house in 1824, Ruffner advertised that he had “opened a house of private entertainment at his commodious residence,” where he would make every effort “to render the lodging of the traveller comfortable, and his diet palatable.” Henry Clay, John J. Audubon, and Andrew Jackson were among those who accepted his offer. Ruffner also invited cattle and hog merchants to take advantage of his extensive pastures and abundant corn. Indeed, his extensive acreage embraced most of Charleston's future East End, including the site of the state capitol and executive mansion.
Holly Grove was reworked after a fire in 1832 and again in 1902. The giant-order Ionic portico, semicircular in plan, that dominates the facade dates from the latter work. In 1979, thanks to interest sparked by West Virginia's first lady Sharon Rockefeller, the state purchased the house and subsequently repaired it.