I think finding him in Norway is going to be extremely difficult. Any possible further info from America could be a big help.
He used the name "Oliver" in America. That doesn't neccessarily mean his name was "Oliver" in Norway. It could have been "Olaves", "Olivarius", "Olivard", "Ollavus", or many other variations. Many immigrants with an uncommon name tended to change it to something used more commonly.
When an immigrant told people he was "from Bergen", it often meant he was from NEAR Bergen, or possibly only emigrated from Bergen. There are many rural parishes nearby that could have been his home also.
Since he married in Chicago in 1875 and died in Illinois in 1891, he should be recorded in the 1880 census with his wife. Finding him would be great....
Do you know anything about his children at all?