Lawrence Eugene Doby was born in 1923 in Camden, South Carolina. His father, David Doby, was a professional horse groom whose long trips to racetracks in the North led to the breakup of his marriage while Larry was still a child. When his mother, the former Etta Brooks, went North looking for work for herself, Larry was put in the care of others, first his grandmother and then his aunt and uncle. While in their care, Doby attended a school in Camden run by the Methodist Church, where he played organized sports for the first time. When he was ready to attend high school, he joined his mother in Paterson, New Jersey. At Eastside High in Paterson, New Jersey, he quickly proved himself a talented athlete, earning eleven varsity letters, in football, baseball, basketball and track. Foreshadowing his early experience in professional baseball, Doby was the sole black player on the school's football team.
Just before his graduation from high school, Doby played his first professional baseball game for the Newark Eagles of the Negro National League—under the name Larry Walker to protect his amateur status. After a year of college in 1942-43 at Long Island University and Virginia Union, he was drafted into the Navy. When he returned to the States in 1946, Jackie Robinson had been signed by the National League's Brooklyn Dodgers. Doby's future abruptly changed. "I felt I had a chance to play major league baseball," he revealed to his biographer Joseph Thomas Moore. "My main thing was to become a teacher and coach somewhere in New Jersey, but when I heard about Jackie, I decided to concentrate on baseball. I forgot about going back to college."