I would consider this family as being you family in 1880. The ages are about right and "Minnie" might be short for "Wilhelmina" for wife and daughter.
1880 Chilton, Calumet, Wisconsin
Charles WINTER, Self, M, Male, W, 60, PRU, PRU, PRU, Farmer
Minnie WINTER, Wife, M, Female, W, 62, PRU, PRU, PRU, Keeps House
Minnie L. WINTER, Dau, S, Female, W, 22, PRU, PRU, PRU, Home
Herman WINTER, Son, S, Male, W, 20, WI, PRU, PRU, Farmer
Additionally, check out Charles Winter in Sheboygan, WI in 1860 if you haven't already. Seems to be the same family as the 1870 Carl Winter in Sheboygan only with a few more children including a 6 month old Herman.