Are you still looking? The records previously posted are confusing to me - surely those records must be referring to the wrong people? Have I got that right?
In order to find the birth parents and birth place of Joseph H Peltier b 1866, the forum needs to first learn as much as possible about him. The idea is to start with what you know, polaris1400, and build from there.
So....would you please post a transcription of your census records and add any other information you have?
If you'll tie information to a specific source, that will help. For example, what is the source that Joseph immigrated to New York? Do you have his immigration record? Do you have a record placing him in New York ca 1867? The 1880 census has a handful of people with that name and that age in New England states, but not New York. Perhaps you only mean that he immigrated in 1867, according to the 1900 census, and that he lived in New York [state? city?] in 1900.
It is important to include late-in-life records as well. Joseph's death might be recorded in a death certificate or obituary; he might be buried at the same cemetery as others of that surname. These can hold clues to his identity in Canada before he immigrated as a baby - presumably with his parents.
The forum can find the records you found, no doubt, but would have to post them here to check with you about whether they were the same as yours. Much easier for you to just post your records to start with.
Thanks so much.