Jarvis Yates passed away in 1883 at the age of 92 and his son, James, took over the farm. James had lived in Kansas in the mid-1850s when there was a great deal of violence between the pro-slavery Border Ruffians and abolitionist Free-State settlers. While there his first wife, Louisa Jane Witts, passed away. James returned to Hatchley with his two children, Harriett and Herbert, shortly thereafter as he was unable to find anyone willing to care for them temporarily.

James married his second wife, Margaret Armour, in 1858. The same year as Jarvis’ passing away James built a large brick house at the southern end of the farm. James and Margaret had five children together: a boy who passed away at birth (1859), John Lyon (1862), Ida Louise (1865), Charles Fox (1867), and Carrie Priscilla (1869). James passed away in 1895.