The corner of N. Churton and W. Tryon streets in Hillsborough was originally demarcated as Lot 98 in William Churton's 1754 survey. The first building on this site was St. Matthews's Church, the Anglican parish church for all of Orange County. A building wasn't erected on the site until after 1768. In 1788, that building was the site of the NC Constitutional Convention. 270 delegates attended the meeting and voted against ratification because the Constitution lacked a Bill of Rights.
A NC General Assembly Bill of 1797 stated that the church was in "a ruinous and decayed situation," and had been untenanted for a number of years. It mandated that the sheriff sell the church and its contents to the highest bidder.
The property then came into the possession of the Presbyterian Church. The town of Hillsborough leased it to them for 99 years at $1.00 a year. A church building was erected on the site in 1816. In 1836, the Presbyterian Church built a Session House in the corner of their property. The Session House was used as a Sunday school and meeting room.
1905 Sanborn Fire Insurance Map
In 1901, the structure was loaned to the newly-formed local library association and converted into the town library.
In 1934, the building was torn down and a new building was built on the site as a United States Federal Civil Works Administration (CWA)/Federal Emergency Relief Administration (ERA)/North Carolina Emergency Relief Administration (NCERA) project at a cost of $15,466.10. The building was named the Confederate Memorial Library by a "local historical group" who donated money to the library in lieu of erecting a memorial obelisk to the Confederacy in Hillsborough. The building had the words "Confederate Memorial Library" in the gable of the front portico.
After the shooting of 10 people in an African-American church in Charleston, SC by a white supremacist in June 2015, Hillsborough was prompted, along with other communities, to re-examine commemorations of the Confederacy. The town council decided to remove the words to reflect the museum’s inclusive nature and support for telling the diverse stories of all residents of Orange County.