Hall was born in Goshen, Connecticut, the son of Asaph Hall II (1800–42), a clockmaker, and Hannah Palmer (1804–80). His paternal grandfather Asaph Hall I (June 11, 1735 – March 29, 1800) was a Revolutionary War officer and Connecticut state legislator. His father died when he was 13, leaving the family in financial difficulty, so Hall left school at 16 to become an apprentice to a carpenter. He later enrolled at the Central College in McGrawville, New York, where he studied mathematics. There he took classes from an instructor of geometry and German, Angeline Stickney. In 1856 they married.