There are at least three apparently related records on Ancestry.
- A convict transportation record indicates that he was convicted on 26 oct 1849 as you recount but that he was not transported until 31 Jul 1852, on the Lady Montagu.
- The court record confirms the conviction date indicates the the Norfolk County Court was then sitting at Walsingham. which would have been in the general vicinity of the offence and by implication quite possibly the vicinity where he lived.
- The 1851 census has him as a prisoner in the convict prison at Portland Dorset - unmarried, age 23, labourer. The Place of birth is transcribed as you report but the original text is far from clear.
You will wish to look at the evidence for yourself and will also see from a map of Norfolk that The Bircham parishes are about 12 miles or so from the Walsinghams (in an area to the north of the main road between Kings Lynn and Fakenham).
This supports the help you have received from 90sum3 and may lead you to further findings. Incidentally, FamilySearch is particularly good on Norfolk PR.
Good luck with your continued research.