Refvik is no longer in Selje, it is in Vågsøy. Vågsøy is both the name of an island and the minicipality (kommune) --the municipality includes a bit more than the island though. The municipality was earlier divided into Nord-Vågsøy (the northern part that includes Refvik, Vedvik, Raudeberg, etc.) and Sør-Vågsøy (the southern part).
I was born and raised in Vågsøy, so if you have any questions about Vågsøy and/or the people who lived there that hasn't already been answered then feel free to ask.