I happened across your post this morning. If you're still interested in Samuel Goff/Gaff, I have some ideas. If you look at the 1860 US census, Dooly Co. GA, p. 524, you'll see that Samuel is listed as having $100 worth of real estate. Unless he had a major setback (which could have happened) it seems unlikely that this is the same Samuel who owned many slaves in Laurens Co. GA in 1850.
Pinkey Gaff [Goff] married Alfred Story 13 Dec. 1863 in Worth Co. GA, Marriage Bk 2W, p. 937. I'm distantly related to this Story family so I know they ended up in Eustis, Lake Co. FL.
There may be more investigating that can be done in Dooly Co. records. Some of the probate records for Dooly Co.are online:https://familysearch.org/pal:/MM9.3.1/TH-1971-30494-11330-43...
I'll be glad to help you look, if you're interested.