In a letter sent to The Times, Dr Hargey said there was "no Koranic legitimacy" for the burka, adding it was "a nefarious component of a trendy gateway theology for religious extremism and militant Islam".
If a woman wants to wear a burqa it is her right to do so.
Baroness Warsi repeated her calls for an investigation into Islamophobia within the Conservative party in a piece in the Guardian, and noted there had been a 26% rise in hate crimes against Muslims a year ago, according to figures from monitoring group Tell Mama.
"Johnson did not go far enough. I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes". They send out a message that Muslim women are fair game. "If you tell me that the burka is oppressive, then I am with you", he wrote.
Meanwhile, former attorney general Dominic Grieve said he would quit the party if Mr Johnson became leader, while ex-minister Anna Soubry said that "many" One Nation Tories would also leave.
He said: "She's clearly wrong to have asked him to have apologised".
"It is crass and it must stop, and it must be condemned by the leadership right from the prime minister down".
Mohammed Sheikh, founder of the Conservative Muslim Forum, said Johnson's article had been "totally out of order".
Johnson is yet to issue a response.
"We have to call it out".