I've been working on James family and this is what I've found so far.
James Howard Smith born abt 1823 County Tyrone, Ireland died 13 May 1888 Amaranth twp., Dufferin Co., Ontario. Married Matilda Hall 19 Oct 1864 Wellington Co., Ont. Not able to find a death record or a second marriage record for Matilda yet. I did track down most of their children all of whom died with out marrying or having children.
I was not able to find Robert Lewis after the 1891 census.
Maria Eleanor "Helen" Smith was the informant on her sister Matilda Dora's death record. Other then that I have not been able to find her after 1891 either. Non of the children below I have yet found after the 1891 census, so I'm not sure when they moved to Northern Ontario. I will keep looking.
Elizabeth Smith born abt. 1865 Amaranth twp., died 20 Oct 1919 South Himsworth, Parry Sound, Ont.
John Smith born abt 1867 Amaranth twp., died 26 Jul 1926 Gravenhurst, Muskoka, Ont.
Mary Smith born abt. 1870 Amaranth twp., died 5 Oct 1918 South Himsworth, Parry Sound, Ont.
James Albert Smith born 12 Jul 1873 Amaranth twp., died 30 Mar 1920 North Bay, Nipissing, Ont.
Thomas Alcorne Smith born abt 1876 Amaranth twp., died 2 Apr 1912 South Himsworth, Parry Sound, Ont.
Maria Eleanor "Helen" Smith born 20 Jun 1878 Amaranth twp., died ?
Matilda Dora Smith born abt 1881 Amaranth twp., died 16 May 1926 Stephenson Twp., Muskoka, Ont.
Robert Lewis Smith born 2 Oct 1883 Amaranth twp., died ?