I recently had a confusing situation - in my case it was a family that had children born in the 1860s who did not appear on any of the United States censuses. I eventually found them on a census in England with birthplaces noted as the United States. Later the family moved back to the United States in the 1890s.
It was helpful to me to use a timeline. For instance:
1876-1877 Event: Move from Marion or Morrow County, Ohio Elizabeth Smith and parents. Place: Sandusky, Ohio
1878 Jan 6 Event: Birth of Maude Sherman Place: Clyde, Sandusky, Ohio
1880 Event: United States Census as Maude Smith, age XX, born XXXX. Place: York, Sandusky, Ohio
Once it is all listed out (and you can put in the mother and grandparents information also) you can get a better sense of what is missing. Put in any newspaper transcriptions, the dates of the witness statements, etc. Very often the witness statements are done decades later and might be suspect. Civil War pension files are full of information, but much of it is recalled after the passing of 20+ years. Ask yourself what you remember of neighbors from 20 some years ago!
The beta city directories might also be helpful.
I have used this technique several times and each time have been able to move on to the next step.
It is quite possible that the children were born out of wedlock. You will most likely never know for certain. I admit it does seem odd for the time era, but take a look at the education levels, professions, etc. Was the move in 1876 to 1877 the result of Elizabeth's pregnancy? You can add this speculation to the timeline. I usually write afte the event: Speculation: move may have been due to pregnancy of daughter, Elizabeth.
Who does Clyde list as his father on his Social Security application form? You might find that information helpful. Have you been able to connect with any of Clyde's descendants?
I usually use a Word document for a basic timeline.