Sorry, I did not see this sooner. I do not do any Baker research anymore. There was at least one Seth Baker living in NY at that time. Mine lived in a place called Pompey then Lafayette and now it is part of Ithica. These Bakers are well documented and are in the Edward Baker of Lyme Mass line. They were some of the first settlers of that area.
On second thought, Seth is too old. He was a g-grandfather after the CW. I would not be at all surprised if your Seth was not named after mine.