Being a farm labourer would not mean he didn't have a house he should be renting/leasing. If he lived on an Estate and worked there he should still be in Griffiths in Westmeath 1854. You see a variety in Griffiths.
Here is an eg:
Bridget Shanley, Parish of Killulagh, Townland Williamstown
(Rochford) tenant of Robert & William Hunt, Poor Law Union/Civil Registration District Castletowndelvin
2d house and garden 0 acres 0 roods & 12 perches
1 perch = 18'x18'
Perhaps Francis/Frank was already in England in 1854 I only have the 1841 & 1871 Census identifying persons born in Ireland.
Thomas Shanley, Parish of Drumraney, Townland Bryanmore Lower, PLU/CRD Ballymahon
1b house, outbuilding and land shared with 1a Michael Cormick, 40 acres 3 roods 4 perches
William Shanley, Parish of Noughaval, Townland Bawn, PLU/CRD Ballymahan, tenant of William Chapman
1A land 3.0.6
1B land 3.1.5
2 land 10.3.20
7c garden 0.0.20 renting this from Michael Shanley
Question here is where is William living no of the above
had a house.