I found a few bits of information on your Houle relatives.
The information contained in that 1880 census was quite helpful.
Julie Houle was née Julie Rivard, daughter of François Rivard and Rosalie Fortier. She married Théophile Houle, son of Joseph Houle and Marie Anne Senneville, on 7 Jan 1835 at the parish of St-Frederic, in Drummondville, Quebec, Canada. They were the parents of at least 12 children, all born in Drummondville. The son listed as Elzin (Elzear) on this 1880 census is most likely the one baptized as Lazare Houle on 3 Jun 1838, having been born on 2 Jun 1838.
As can be seen from the census, the three oldest of Elzin's children were born in Canada, and the 3 youngest in Maine.
Elzear Houle's first marriage was to Emelie Hamel on 24 Jun 1862 at St-Hippolyte, Wotton, Richmond County, Quebec. She was the mother of the three eldest children.Emelie Hamel, wife of Elzear Houle, died 28 Jul 1870 and was buried 30 July 1870 at the cemetery of St-Hippolyte, Wotton, Richmond County, QC. On May 25, 1872, Elzear Houle married Sophie Rioux in Biddeford, Maine. Sophie Rious was the daughter of Leonard Rioux and Perpetua Labbé.