You confused me for a moment because I didn't know the Rev H Smith had a son named James Bey Smith, I knew only that he had a brother of that name!
I don't have much on the Smith side of your family, I'm afraid, just back as far as John Smith b.1797 at Tarland and his wife Margaret Bey b.1803, parents of Harry Smith b.1830, who would be your g-grandfather. And all of that info I got from the ancestry tree of June Riddell, who wrote some of the earlier messages in this thread. So if you click on her username you should be able to access her tree from her profile page.
You and I are both connected through Elizabeth Stewart. Her younger sister Margaret was my gg-grandmother. Harry Smith was a witness at Margaret Stewart's wedding at Allalogie, Logie Coldstone in 1869.
Elizabeth and Margaret's parents were James Stewart b.1797 Mortlach, Banff and Elizabeth Blair b.1803 Crathie, Abdnshire.
James Stewart and his brother George were declared fugitives on 19 September 1816 and 'put to the horn'. Presumably that's why James moved south to Aberdeenshire!
James Stewart's parents were Robert Stuart alias Macalay b.1754 Mortlach, Banff and Margaret Ritchie b.1752 Mortlach. Quite a few folk in the Banffshire area were in the habit of having an alias added to their surname, and one theory is that the alias gives an indication of the geography or affiliation of a clan surname. The MacAlays were a sept of the Stewarts of Appin.
If you need more info on this side of your family, just message me and I'll send you an invite to my tree.