There is a connection of sorts for Lem W (showed on census as Lena W) Mallory and your Mallory line. Lemuel's father, Ward Mallory and his uncle Samuel Mallory both lived in South Lansing NY for a time...and owned a tavern there. John Mallory, brother of your Elijah, and husband of Sarah Richardson also lived in Lansing NY (South Lansing was called "Malloryville.") Ward and Samuel both moved to Wisconsin and brother Henry to Michigan which is where John Mallory and Sarah Richardson ended up.
Henry's grandson George Van Ranselaer Mallory b 1899 Kent MI d 1976 Florida is actually buried at Hazel Ridge Cem in Elkhorn WI with Samuel and Lem's brother Henry L Mallory b 1840 Ny.