The British actor and comedian has claimed allegations against him are part of a wider ‘agenda at play’