I believe you were right. The Amenia Historical Society replied to me:
Amenia did have a log jail during the Revolutionary War. Amenia may be about 25 miles from the River, as the crow flies. Amenia is about 35 miles from Hudson, NY, and about 35 miles from the River at Poughkeepsie. I do not know if there were other log jails in the county.
Early History of Amenia by Newton Reed, published in 1875, has a lengthy chapter on the Revolutionary War, including some of the activities of the 6th NY Regiment, and mention the log prison. He stated that the rude structure was located about 1/2 mile east of present day Amenia, and north of the turnpike, as the road was called in his day. He said that the prison was used for Tories and other suspected individuals. The remains of the building were still in evidence in the early 1870's.