I have read St. Croix census of 1841, 1846, 1850, 1857, 1880, 1918 and abstracted all I could find pertaining to the Irish. I probably missed some entries but did find the following Irish Smiths:
______ Smith, age 24, born Ireland, baptized Church of England, overseer on Mt Pleasant Estate owned by R. S. Grant according to 1857 St. Croix census.
Richard Smith, age 25, born Ireland, Episcopalian, overseer of Fredensborg Estate owned by Barings & Co., in Kings Quarter according to 1857 St. Croix census.
There are several Latimers in my notes:
______, Latimer, born Ireland 1831, baptized English Church, manager of Spring Garden Estate on North Side, owned by Wm. Moore, Esq. (1857 St. Croix census)
Alexander Latimer, age 50, born Ireland, settled 1851 in St. Croix, Episcopalian, planter/owner of William Estate in North Side Quarter (1880 St. Croix census); also in household is Mrs. Latimer, M. A. Latimer, age 32, born Ireland, in St. Croix since 1871.
George Latimer, age 25, born Ireland, settled in St. Croix 1877, baptized Church of England, overseer on Estate St. George and Wal___ (1880 St. Croix census.)
James Latimer, age 40, born Ireland, settled in St. Croix 1860, Episcopalian, planter/owner of Estate Whim owned by A. and James Latimer (1880 St. Croix census).
John Latimer, age 30, born Ireland ca1820, Protestant, manager of Castle Coakley in Queens Quarter (1850 St. Croix census.)
Mary Ann Latimer, age 70, born Ireland, planter of sugar estate at West End Quarter according to 1918 St. Croix census; also in household is Thomas Latimer, age 57, born Ireland, manager of sugar estate.
Thomas Latimer, age 20, born Ireland, settled in St. Croix 1879, English church, unmarried overseer of Estate (appears to be Hankyon or Thankyou) according to 1880 St. Croix census.
William Latimer, age 24, born Ireland 1878, Episcopalian, overseer on Estate Camp Rico, 1880 St. Croix census.
There you have your James Latimer, shown as planter/owner of Estate Whim in 1880. You need to get a step or two back to connect your Smith family with Latimer. Find your grandfather's death record - the most important step!
How wonderful to have a connection with Whim! I was there last June and again this February. Contact the Whim library for Latimer data which should be contained in the Whim Estate file (estate file collection). Some of those files are almost empty, others contain a wealth of information.
Perhaps Whim has cemetery records or church records for St. Paul's Episcopal Church, that would give you more clues on your Smith line.
Keep in touch. Regards, Norma Meier (Illinois)