Here is what I found:
1910 United States Federal Census
about James E Payne
Name: James E Payne
Age in 1910: 52
Estimated birth year: abt 1858
Relation to Head of House: Head
Father's Birth Place: Virginia
Mother's Birth Place: Virginia
Spouse's name: Louise
Home in 1910: Richmond Henry Ward, Richmond (Independent City), Virginia
Marital Status: Married
Household Members: Name Age Color Job
All able to read and write
James E Payne 52 Black Coachman for private family
Louise Payne 45 Mulato Housewife
Mary Bell Payne 22 Waiter Lunchroom
James E Payne, Jr. 21 Porter Road house
Richard T Payne 20 Tailor Tailor shop
Has both married 23 years. He has a M2 and she has a M1 by their names. I don't know if this means it's his 2nd marriage or not. In 1900, all children were in school. The home was rented.
Have you done the Payne DNA search? We did with my Daddy, James Ernest Payne, b. 1916-2005, but are in the Unknown category. Who is to say that we couldn't be kin way back when. I'm sorry to say that the Paynes of VA and KY did have slaves. My lineage stops at 1804. We can't find anyone to claim or documents that match Elijah F. Payne.
Good luck in getting past your brick wall.