I don't know if my James HOLDER is linked to the one you mention, but here goes:
my paternal great-great-grandfather James (__) Holder was born October 1811 in Virginia, but by 1860 he was married (to Margaret Humphrey(s) ), working as a blacksmith, and living in Scott County IL. He died March 15, 1881 in Johnson County MO. His children include Sarah, Matilda, John, Martha, Alice, James, Eliza, Martin, Lydia and my great-grandfather David William Holder.
I hope something here rings a bell, or helps in some way. Am flying a bit blind.