My thoughts... whatever her actual marital status when she married Edwin Manners, Sarah would have stated she was a widow. Either she WAS a widow (and you still need to find Henry's death), she was divorced (very unlikely as you say due to expense) or she and Henry were separated - she would claim she was a widow, not that she was still legally married to someone else!
Baby Henry's birth was in late 1900 and she is on the 1901 census with a 4 month old baby but no husband. As baby Henry's father's details showed him working in London at the time of his birth and he is not around 4 months later, it is possible that they separated before or shortly after his birth and Henry made a new life for himself in London.
Sarah would most probably have been able to remarry on her say so that her first husband had died. Interestingly on the 1901 census her marital status was first entered as M (married), this is crossed out and replaced with S (single). If she was a widow in March 1901 that would most likely have been entered as her 'occupation'. The change from M to S would suggest to me a separation.
1. Either Henry snr died between baby Henry's birth (assuming he is not showing as deceased on the birth certificate) and Sarah's remarriage... or
2. Henry was still alive and well and living in London when Sarah remarried; therefore you could consider looking for Henry on death records and/or census records in London. He could even have remarried himself... or
3. Henry's death transcript has his surname misspelt, eg, as Janes, James, Sones etc so nothing comes up when you search 'Jones'; you could consider looking for likely misspellings (in Swindon or London)... or
4. Henry may have abandoned Sarah before baby Henry's birth and for the sake of her pride/respectability she claimed he was working away; the Woolwich Arsenal reference may have been made up (possible as this is the only reference you have to it). What was his occupation as shown on his marriage certificate to Sarah in 1898?
I hope this is of some help. Good luck!!