Not sure if we are related but have a Great Great Grandmother, Margaret Roarty, from County Donegal. She married a Peter Gallagher from Donegal as well. I do not believe they came to America but their son which was my Great Grandfather, William Gallagher, definitely immigrated and lived in Benwood, West Virginia. I am not sure about the number of Roarty's in County Donegal but Margaret would have been born between 1830 and 1840 which is similar to your ancester James Roarty. I will continue my research and let you know what is discovered.
All the best!