Would you happen to be any relation to these guys:

On 1850 Census, Rowland Williams resided in Castleton, Rutland County, Vermont with his wife, Elisabeth, and children: Rowland, Robert and Ellen. David Morris (age 21 born in Wales) and Edward Morris (age 19 born in Wales) were also living in this dwelling and working as quarry men. They are not listed at the end of the family members on the census as non-family members. They appears to be listed in chronological order by age with the other family members.

Interesting to note that my 3rd great grandfather was the head of the household Rowland MORRIS Williams. Just wanting to follow through on a hunch and long shot.