I, too, am a descendent of Thomas Ayer and my lineage is the same as yours through Alexander Hamilton Ayer. My grandfather was Henry Mason Ayer, brother of Charles Petit Ayer. Although I am a rank novice in this business, the best information I have found about our family is Louise Ayer Vandiver, Some Data About the Ayer Famikly in the South. I have only a xerox copy of it and have no idea when or where (or even if) it was published.
What might be of interest to you is that I grew up in the home of my Grandfather Henry Mason Ayer, and not more than a mile or two from the site of Alexander Outlaw Ayer's home when he moved ffrom Tennessee to Kentucky--in about 1833. Several years ago an uncle took me to the site of the Alexander Hamilton home, where our grandfathers grew up. That is also very near where I grew up.