I have also heard that Jelsma is a derivative of Jelle and -sma (which would be translated to mean son of Jelle). However, I don't think Jel means yellow in Dutch. The Dutch word for yellow is geel.
Also, most names that were derived from places end in -ma. That may be another reason why it is believed that both Jelsma and Jelsema are derived from the place Jelsum.
I can't say for certain which of these is correct (or if both are correct depending on which Jelsma line is being researched). I do know that I have not found any Jelle names in my direct Jelsma line. I have mostly Harmen & Hendrik as first names.