You are correct that Ellis was not with Harriet, but it is the 1880 census and not the 1920 census that I mentioned. Ellis was indeed with another family, but I couldn't make the connection. And, Harriet was with another family...a whole bunch of other surnames.
However, I still think that Harriet was the mother of the three boys. Hampton and George both had daughters named Harriet. Then, today I actually found Harriet living with George and Isabella [1870 census].
Lafourche Parish [page 493]
George Morgan 21 [picked up as Mugan]
All four are shown as being from LA, but that may be incorrect for Manuel and Harriete. Also, Manuel's age may be incorrect. In 1880, George was 24, but should have been more like 31 and Isabella was 26 and likely should have been more like 30. So, George was likely born ca 1849. This Harriet was born ca 1824 and the one in 1880 was born ca 1825. I think it is the same person.
In 1870, I found Hanton Morgan 25, Octavia 19, and Adel 1. With them was Octavia's mother Dolly Graves 55. In 1880 she is listed as Burrell. They were listed on page 496. Also in 1870, page 534 was Manuel Morgan 59 NC and Lucy Manual 35 SC. Manuel may have been the father of the other Manuel. Lucy is too young to be his mother. I don't know if she was Manuel's wife or daughter.
In 1870 though I found Hampton and George, I did not find Charles.