Sorry I don't have info on your Joseph but I think we might be searching for the same info.
I am searching for Samuel Baker who married Betsey Trent n 1796 (Bedford Co., VA) They had son, Elijah who married Nancy Gross in 1819 (Bedford Co., VA). However, Sam is listed in the 1810 Botetourt Co. census and both Sam & Elijah are listed in the 1820 Botetourt Co., census. Maybe, your Joseph is brother to my Elijah?
There is also a Robert & Henry Baker listed in the 1810 Botetourt Co., VA census which would put them at the right ages for Sam's dad.