A Herefordshire couple claim that three former barns that were converted into holiday lets have in fact been let out as homes since the Noughties.