I did look through 1841 births for all the churches in Begen. There are no records for a person named Oliver born that year in Bergen. I really feel he was born NEAR Bergen, not IN Bergen.
I think the only way to finds him is with an exact birth date. You really need to get a copy of his death record to see if there is more info on it, possibly it will list an exact birth date or place of birth.