Toss that rumor of the covered wagon. Definitely untrue. Per his WWI Draft Card, Benjamin Oliver was born in Brazos County. Your tree displays Dr. William Holt Oliver (18 Feb 1868 - 7 Feb 1933) as Ben's father. That's also untrue. What is true: James P. Oliver was the father of W. H. Oliver. I don't know the year Ben settled in Waco but it had to be after 1940. Travel via covered wagons was no longer the primary means of transit. Willie Oliver (Ben's father) wed "Elizzie" Franklin 24 Dec 1888 in Brazos County. That's found https://familysearch.org/
You have the family's 1900 Brazos County Census. Several of the children relocated to, and died in, Galveston County. However, the parents remained in Brazos: 1910 Precinct 1-District 1-Image 21, 1920 Precinct 1-District 1-Image 6, and 1930 Precinct 1-District 2-Image 18. Having trouble finding death records for both parents.