The site was originally part of the medieval manor of Priorshold. The plot of land and the house were leased to Wantage tradesmen after the dissolution of the priory in 1538. Archives of the Dean and Chapter of Windsor show that the site was let to Alexander Doe in 1662 and then to William Hazell, a wheelwright, in 1683. Lawrence Hazell, a clothmaker, was responsible for the major rebuilding of the house in 1780, commemorated by the date stone bearing his initials over the central first storey window. In the mid-19th century the house became the doctor’s surgery and remained so until 1974 – hence its present name, the Old Surgery.
In the mid 1970s, the newly-formed Vale of White Horse District Council bought the site and leased it to the museum trust. The Vale & Downland Museum Centre formally opened in 1983 and major improvements to the main gallery were completed in 1999.
In 2011 the owner of the adjoining Legges Cottage which housed the kitchen and archive store, decided not to renew the long term lease on the building. Following a fundraising campaign, a new extension has been built to house a new kitchen, office and archive store and to extend the upper gallery.