I think that the story you might be thinking of is the Lee family scandal that involved Henry Lee IV, nicknamed "Blackhorse Harry", the eldest son of Governor and Revolutionary War General Henry "Lighthorse Harry" Lee from his first marriage to his cousin Matilda Lee, the heiress of Stratford Hall.
Under the terms of his late wife's will, General Lee inherited a life tenancy of Stratford Hall but after his death it would revert to their son, Henry ("Blackhorse Harry").
As well as inheriting wealth, position and one of the great plantation of Tidewater Virginia, "Blackhorse Harry" also inherited his father's inability to manage money wisely. Unfortunately, he did NOT inherit his father's strength of character, sense of honor or ethics.
"Black Horse Harry" married Anne Robinson McCarty, daughter of Daniel McCarty a wealthy local landowner. As Mr. McCarty was dead, and his widow Margaret Robinson McCarty (remarried to Richard Henry Stuart of Cedar Grove) had also died in 1808 in childbirth, "Black Horse Harry" was made the legal guardian of his wife's teenaged younger sister, Elizabeth "Betsy" McCarty.
With his gambling and high living, he rapidly ran through both his wife's and his ward's inheritances. Worse, he took advantage of his teenaged sister in law and got her pregnant.
If you are seeking descent from a misplaced Lee, farmed out to a local family to be raised to avoid embarassment, this might be the source of the original rumor. However, history records that this child was stillborn.
Tragedy was shortly also to visit his wife, Ann McCarty Lee. Their young daughter, born in 1818, died at the age of two after falling down the stone exterior stairs at Stratford Hall. Legend has it that the ghost of this child sometimes appears at Stratford.
Mrs. Lee, brokenhearted both at the loss of only daughter and the scandal of her husband's dishonoring her sister, left her husband and went to Tennessee, where she developed a laudanum problem.
"Black Horse Harry Lee" outwardly unfazed by the scandals and tragedies, continued in his previous lifestyle, but found that he was shunned by many. Never afraid of controversy, he was one of the first "political hacks" in America. He wrote a political pamphlet that "outed" the rumor of Thomas Jefferson's affair with Sally Hemmings. Lee's detractors, understandably, charged that he'd done this merely to deflect attention from his own scandals!
By 1822, Lee had managed to run through his own fortune and those of his wife and sister in law. He was badly in debt, primarily from gambling. As a result, he had to sell Stratford Hall and went abroad. He lived out the balance of his life in France, where he befriended Napoleon's aged mother. His wife eventually kicked the drug habit and they were reconciled. She joined him in France where he died in 1837 and she died in 1840.
In 1828, Stratford Hall was bought by Harry Dent Storcke.
Now THAT was poetic justice.
Mrs. Harry Dent Storcke was none other than the former Betsy McCarty, the dishonored teenaged sister in law of Black Horse Harry, who had taken her fortune and her reputation. But she "got her own back" in the end.
However, not all could be restored to her. She was never again able to have children, and wore mourning clothes from the time she had lost the baby until her death in 1879.
Mr. Storcke had predeceased her, and as they left no direct heirs, Mrs. Storcke willed Stratford Hall to her two half brothers from her mother's second marriage, Charles and Richard Stuart.
Ironically, the latter was the baby born in 1808 when their mother died in childbirth. Had their mother not died, perhaps young Betsy would not have needed to have a guardian appointed, or perhaps her mother's second husband would have filled the role, thus keeping her from the clutches of her libertine brother in law.
This is the stuff of which novels are made!! But it's a true story.
The scandal was still sufficiently fresh in people's memories by 1830-31 that it almost caused G.W. Parke Custis and his wife Maria Fitzhugh Custis (herself born in King George County) to forbid their only daughter, Mary Randolph Custis, from marrying the handsome young Lieutenant Robert E. Lee, although they knew his own character to be above reproach. But eventually relented and the rest is history.