The modular steel-frame, concrete, and glass-block building colored in black, white, gray, soft brown, and bright green meets two perpendicular streets. An irregular entrance wall faces the curving approach road. The center was intended to provide community recreational facilities within walking distance from downtown that would stimulate housing development in a cleared urban renewal district that displaced African Americans in the heart of Detroit's Black Bottom neighborhood. First known as the Gratiot Redevelopment Area, the site was the subject of a master plan in 1954 by Oscar Stonorov, Victor Gruen Associates, and Leinweber, Yamasaki and Hellmuth. The project languished from 1951 until the plan was done and for twenty years after the Lafayette Park section of the site was completed in 1959–1960.
The two-level interior is flooded with daylight and is arranged with zones for arts and crafts for senior citizens, community purposes, and recreational activities including a natatorium, gymnasium, handball courts, boxing and exercise areas, and community meeting rooms. Activities are grouped around large lobby spaces.