Just a quick check of census reveals the following could possibly be the family of James' father, Elijah C. Buff
Elijah Buff parents: Martin Buff and Nancy Lea/Lee, married 18 Sept 1839 in Haywood Co., NC
In 1850, in the home of Martin Buff is Catherine Buff, age 52. I would assume this is his mother. (I don't know without doing futher research if this in Catherine Lyons, but that name is ringing a bell in past research I have done and/or read)
I did not spend alot of time doing detail researh and comparison, but his should give you some direction.