Thank you for your message. There are many bells ringing in your information. I had to check a few details with my mum's cousin who is also researching the Bunces which why I didn't reply straight away.
We had established that John Bunce originated from Great Marlow but had not found any baptisms etc to support this. However Sue does have John's Indenture certificate into the art of cordwaining from 1811. This states that he was a labourer at Great Marlow and it was paid by his (great?) uncle, Thomas Bunce. Varous other details you mentioned fit with our research there. We hadn't found John's sister Caroline though, so that is very interesting. May I ask where you were able to find this information because we have been searching for ages to find John's parents.
We also knew about Louisa marrying Thomas Gaylard. I wasn't aware of the Unitarianism but I think Sue was. Thomas Gaylard seemed to officiate over several marriages and funerals in the family.
If you would like to discuss this further my email is firstname.lastname@example.org