Isham Lee Baker was born in Tennessee in February of 1833 and was listed with his parents on the 1850 census of Tishomingo County, Mississippi. The family moved to Gonzales County, Texas in 1849-50. Isham Lee is listed in the 1860 census of Gonzales County with his wife, Mary Shoemaker, and three sons: Abraham, Jesse Harden, and James Henry. 

Isham Lee served in the Confederate Army from 1861 until the close of the war in Company C, Ford's Regiment, and/or Company A (Captain Ford's command) of the Benivedas Regiment. In 1863 he was at Ringold Barracks and Brownsville along the Rio Grande. 

After the war and Mary's death Isham Lee moved to north Texas/Indian Territory, apparently to avoid signing the "Iron-Clad Oath" of allegiance to the Union and to avoid living under Federal Military Rule in Texas. Isham Lee and Mary had five children, four boys and one girl.

Isham Lee married Jane Hughes in Colorado County in 1868. They had one son, Russell. Russell was listed in the 1880 census as living with his uncle, Jesse H. Baker, along with his half-sister, Ida Jane Baker, in Cooke County, Texas (along the Texas/Oklahoma border). Jane Hughes' fate is unknown.

Isham Lee married Mary Jane Needham in 1871 in Ellis County. They had twin girls in 1873. One of the twins and Mary Jane died soon after the twin's birth. The other twin, Ida Jane, was raised by her uncle Jesse in Cooke County until 1884 when she moved to Jack County to live with her half-brother, Abraham.

Isham Lee married Phoebe Elizabeth Boggs in 1874 in Cooke County. Phoebe had a son, William, by a previous marriage. William took the Baker name. Isham Lee and Phoebe had seven children. They moved back to Gonzales between 1877 and 1881. The last child was born in 1888 when Isham Lee was 55. 

In total Isham Lee was married four times and had 15 children. In 1902 Isham Lee applied for a Confederate pension. In the application he stated that he had lived in Gonzales County (Smiley) since 1849 except for nine years after the war. At 72 he had rheumatism and kidney problems (enlarged prostate), "two head of horse worth $00.00", and no land. He died in 1909 and is buried in the Bundick Cemetery, Smiley, Texas.