Jesse Hoyt (1815–1882), a New York investor in Saginaw timberlands, left to East Saginaw the gift of a site and $100,000 with which to build and operate this library and to purchase books. In preparing and choosing plans for the building, the library trustees visited several midwestern libraries and enlisted the advice of Frederick Poole, who was head of the Chicago Public Library and who later was elected president of the American Library Association. Poole believed that a librarian should plan the interior of the building and that an architect should design the exterior. Van Brunt and Howe of Boston; McKim, Mead and White of New York City; and H. H. Richardson of Boston responded to a competition for its design and submitted plans. In 1886 the trustees rejected those plans prepared by Richardson, America's leading architect and a foremost designer of small libraries, in favor of those drawn by Van Brunt and Howe, presumably because they followed Poole's instructions. According to Poole's plan, the library was to be cruciform in plan, utilitarian, and with side rather than sky light, and the reading room lighted from the north. Richardson's design for the Hoyt Library was adapted in 1886–1888 by his successor firm, Shepley, Rutan and Coolidge, for the Howard Library in New Orleans.
Although not by Richardson, this building with its round corner tower, arches, intersecting gables, and rough stone exterior clearly reflects his influence. The exterior walls are constructed of randomly coursed grayish-white limestone, quarried locally at Bay Port, and are trimmed with red Portage Entry sandstone. The interior is finished in oak. From the vestibule and hall off the original round-arched entrance portico on Jefferson Avenue, which is now closed, a stairway ascended to the original book delivery room and the main portion of the building. The original reading room fronts on Jefferson Avenue. The second floor contains the upper portion of the book room and a lecture room, stage, trustees' room, and room for special collections. The Norman porch decorated with arches and columns of carved red sandstone was modeled after Canterbury Cathedral in England and added in 1920–1921 on the Janes Street side of the building. It became the main entrance. This addition includes a large reference room and stacks and the conversion of the main reading room to a children's room.