In the 1860s there were few permanent inhabitants of the area now occupied by Buford.  Silas King and his family lived in what is now downtown Buford.  William Sudderth owned land to the south and east of King. Wyatt Wilson owned more than 1,000 acres north of Buford. Other early