OK Lindy. I finally found Hugh(born 1853)in the 1880 Census your side.
He is in Lyndon, Caledonia, Vermont. Age 27 and he is a
brakeman on a train which probably answers your query on how
he ended up in Duluth by 1885 or so.
His parents were both born in Ireland.
Do you know if Hugh had any brothers or sisters???