Does this seem to be the family in 1880?
https://familysearch.org/pal:/MM9.1.1/MN47-RY8

You can find other census records and vital records for free at the www.familysearch.org site.

In 1870 the family is in Milwaukee, Wisconsin
in 1900 James, Elizabeth and Charles are in Green Bay, Brown County.
I would suggest you post a message on the Brown county board
here http://boards.rootsweb.com/localities.northam.usa.states.wis...