I believe that most Tillou/ Tilyou families in America were descended from Pierre Tillou born 1655 in France. He was a Huguenot and had to escape from France (Saint-Nazaire). He sought sanctuary in England and then sailed to Virginia. His descendants settled around New York. The name Carlisle / Carlye appears often in the Tillou / Tilyou line. One of Pierre Tillou's descendant's , Pierre Joseph Tilyou, married Mary Carlisle. They lived in Dutchess County, New York. I believe this is where the name Carlisle started to show up. So I would think that your Carlisle was descended from the two people as my Carlisle was but in a different line.