The only way to find out if he was in an orphanage would be to track his locations in the various census (1881, 1891, 1901 and 1911).
I thought I would try and see if I could trace Walter Walsh born 1881 through the censuses. I started searching on a broad basis, and spotted in the 1881 census a Walter Walsh aged 19 who was born in Ireland and living in Pontypridd (which is not that far from Pontypool), and wonder whether there is any connection to your family. (Census refs: Piece: 5300; Folio: 48; Page: 25).
Going back to Walter born 1881, I can’t find him in any census until the 1911 Census, when he is married and living in Southall, Middlesex, (which is today part of greater London) and working as a chimney sweep (Census refs: Piece: 6800; Schedule Number: 59). Also there is his brother Owen. The census says that Walter and his wife have been married for 8 years, and have had 4 children, but only 1 is still living.
Walter David Walsh married Clara Louisa Wilkinson in Kensington, London in Qtr Apr 1903 (Vol. 1a Page 366). The parish record which Ancestry have says that his father, William, was deceased, and had been a “cab proprietor”. Clara’s father is not listed, but there are 2 Wilkinons who were the witness. Neither appear on the 1901 census with Clara. Walter and David were living at the same address and as Clara was only 17 she would have needed her father’s permission!
Clara is listed in the 1901 Census with her parents and siblings in Islington, London. I am now very confused as in the 1891 census, she is listed with different parents, who could well the witnesses to her marriage… (Piece: 38; Folio: 181; Page: 34).
I can’t find wither Walter or Owen before 1911. Could they have been brought up in Ireland?