Me again Mary with a tad more. Your Joseph is the lad married to Madeline Comeau in 1876 and your Elizabeth (1884) is named after her grandmother on her mothers side.
I think this is Joseph and family in the 1881 Census http://data2.collectionscanada.gc.ca/e/e325/e008120598.jpg
See Maggie and I believe this is the same Joseph in the 1891 Census, note dau, Elizabeth aboard.http://www.collectionscanada.gc.ca/databases/census-1891/001...
I'm just new with Genealogy, so I could be in error, you know.