This family is on the 1861 census
Upper Road Row, Huddersfield
James Barratt Aged 67 Stone Labourer b Co Mayo
Mary 48 b Co Mayo
James 22 Stone Labourer b Almondbury, Yorkshire.
Anthony 18 Stone Labourer b Ireland
James (son) was born at Almondbury, which is on the outskirts of Huddersfield. It looks like Anthony was born in Ireland, so perhaps James & Mary were going back and forth on visits to relatives there.