Eliza Jane Rugg is d/o William S.Rugg, b. 1819 VT, and Hannah A. Hunt, b. 1824 NY. Elmer is living with his grandparents in 1880 at Linn, Walworth Co., WI. Hannah died in 1901 and Wm.in 1903. Both are buried in S.Hebron Cem, McHenry Co., IL.
1900 census Westerville Twnp., Custer, NE lists a Elmer "Damer" living with James & Lilly Baker. He comes close to the description of your Elmer.
I am connected to the family through Eiza's sister Mary Louisa, b.1845 NY. She married Alhanon W. Johnson, b. 1841 NJ.