Bishop Williams, a farmer’s son, was born in Richmond, Virginia, on July 18, 1829, and brought up in straitened circumstances by his widowed mother. He attended the College of William and Mary and the Virginia Theological Seminary.

Ordained deacon in 1855, he offered himself for work in China, where he was ordained priest in 1857. Two years later, he was sent to Japan and opened work in Nagasaki. His first convert was baptized in 1866, the year he was chosen bishop for both China and Japan.