I have asked some of my friends and as far as they are aware, if a person doesn't show up on the indexes there are several options;
He could be indexed under different spelling. (Howard, Heyward, Heywood, etc etc etc.)
He could be in a far flung place like Scotland or even the continent and not in the index.
He could be indexed under his initials only.
He could have been simply missed.
He could be on a ship/boat that was not put onto the index.
Or any number of other reasons.
I think that the obvious thing to do next is to look at Caistor in 1861 & 1871 as well as 1891 and 1901. 1901 is on line (go to Yahoo and type in 1901 census, click on the pro.gov one that comes up. I have had a quick look but couldn't find Herbert Hayward.
I looked at the 1851 census index for Norfolk but could not find Herbert Hayward either but i did find an Annis;
Robert George H Fisherman & Salvor Bn Caistor
Maria ,, W ,,
Annis ,, D 2 ,,
William Squires B in L ,,
These details all need checking as they are from an index.
Go to your nearest LDS Church and ask where the nearest LDS Library is, then you can order the 1861, 71, 91 census on film.
If you have any problems get back to me.