The Caldwell Block was built for either Alexander or Edward (brother) Caldwell in 1857. From 1909 to 1910 the Gem Theatre, Whitby's first moving picture theatre, was located here. After 1910, stores occupied the main floor until the 1920's. In 1959, the Independent Order of Odd Fellows (IOOF) occupied the building. During demolition, it was discovered that the original brick structure was bricked over by another layer of brick. This is estimated to have taken place sometime after the turn of the century, based on the style brick. The original IOOF sign was saved and donated by Wade Szilagyi to the Lynde House Museum.