Elisha K. Williams you're referring to is my great great grandfather. His son, William Walter Williams is my great grandfather, whose son Marion Logan Williams is my grandfather.
Elisha was the one that gave me the hardest time getting past as well, but finally located him. His father was Walter Watson Williams and his mother was Millie Green. Walter was born c.1801 in what was Rutherford County, N.C. at that time. He died c. 1888 in Henderson County, NC.
Milly Green was born c 1804 (no death date for her), and her parents were Amos Green and Elizabeth Searcy.
If you need more info, please let me know. I'd love to compare notes to see if you have info that I haven't located as well.