I found the record for James Sparrow in an overall search for Rhoda Pearce in North Carolina. And I did read it wrong - it is definitely David. I wish I could blame it on me needing new contacts, but I just had my eyes examined a few months ago...I think my brain turned it into Daniel because the one time I found Robert Moore, he was with a Daniel Moore in the same household. David Moore could definitely be your David Pearce.
The only other thing I can think of right now would be church records. And even then, those would probably just give you more of an overview of David's immediate family, and not his parents. But, finding out who he was close to at church might help you find more family stories.
As for the earlier census records and the free women of color being listed as heads of household, I think it's more a case of the fathers being slaves, as opposed to them working away from home. Even if a man was working somewhere else, the census was supposed to reflect place of residence. One of the problems with trying to figure out a neighborhood in the pre-1850 census records is that free people of color were supposed to be listed on a separate page, so you can't see who the neighbors are. If you could see the neighbors, then you could check to see if any of them were slave owners, and you can do that with the 1850 and 1860 years.
I've got digital images of the Court of Pleas and Quarter Sessions minutes for Tyrrell County, as well as images from three boxes of slave and free people of color records for that county from the NC Archives, so I can say that based on Tyrrell County, some slave owners had no problem with allowing their slaves to marry and would issue passes so that the men could go to visit their wives. In those cases though, it's not mentioned if the wives are slaves on other farms, or if they're free women of color. I know of at least one case where a slave owner had a complain against a white woman because one of his slaves was living with her, and he wanted her to leave his slave alone. Looking at the bastardy bonds around that same date, she was also having kids with this man, but unable to marry him because she was white.
Another thing to be aware of, is that some of the family lore might be wrong in order to save family pride. I've been helping a woman with her family, most of which were in Bertie and Martin, but some in Washington County, and the thing that her mother and aunts were most adamant about was that they only had one ancestor who was a slave - everyone else was a free person of color or Native American. After 4 years of research, we've yet to find a free person of color in her family tree. Ticked the mom and aunts off a tad bit, but the records don't lie. So, it could be that Charles Moore was Native American, or it could be that he was a slave. Either way, as a person of color he and Rhoda would not have been allowed to marry. But things like that are the things that acquaintances, especially ones known for a long time, might tend to gossip about, so finding David's friends and acquaintances, and then locating their descendents, might get you more stories and/or a different perspective on David's story.