If your line of Carroll family is anything like mine it is because most of them were outlaws to begin with. Much like Charles Carroll of Carrollton who was an outlaw in British eyes.
Like most everyone here I have been told about our family connect but I believe I will have to go back beyond Charles and then work my way forward again. Best of Luck to you but if I run across any James "E" in my line I will let you know. I have both a James "H" and a James "F" both in the KY/IN area thus far.