Don't forget it wasn't strictly 'emigration' in the formal sense as it was all one country.
You don't specify which part of Ireland he or his parents came from or what their occupation or educational standard was...
There were regiments stationed and raised in Ireland (as there were and still are in all parts of the UK) so that could be a reason.
Another could be that the original Mr Clifford came over to work - maybe a sailor or he worked for one of the Irish aristocracy who had homes in England and Ireland. He may have worked for the civil service in Dublin
Also - prior to the 19C there were few universities in the UK - Trinity College at Dublin, would alongside the 4 in Scotland , have been a destination for English Catholics and non-conformists (who were barred from Oxford and Cambridge)
Just a few reasons why an Englishman might have moved to Ireland (even if temporarily)