Also opposing the move, Conservative peer Lord True said Mr Johnson had "never said" he would suspend Parliament, and "outrage" about the possibility of this occurring has been "got up" by "Remainers".
That means lawmakers would be sent home two weeks earlier, hindering their ability to stop Britain leaving the European Union without a deal.
Conservative Party members are presently in the final stages of selecting the next leader of their party, who through the composition of the present British Parliament will become the next Prime Minister when the votes are counted and the succession made next week. Mr Johnson then pulled out of the contest.
He added that Mr Johnson failing to rule out the measure would be one of the defining issues of whether he could offer him any support.
Ian Murray, an opposition Labour lawmaker and campaigner for a second referendum, said he anxious about the process to select a new prime minister, with only Conservative members making the decision.
The EU could offer London a "bare-bones" free-trade deal and commit to working right after Brexit on alternatives to the backstop, a way of avoiding extensive checks on the Irish border that is vehemently opposed by many in Britain.
Their stance was a blow to middle-ground MPs who believe the European Commission would agree to such a compromise, and it was seen to increase the chances of a no deal.
- Why might a new prime minister prorogue Parliament?
Mr Grieve said it is an "astonishing" and "disgraceful" suggestion from the man most likely to be the next PM.
"I was surprised by what they both said and I think their views will collide with the reality when whichever one wins, starts negotiating and starts dealing with a Parliament which may be more hard than they think to engage with".
Mr Bebb appealed: "My colleagues in government need to think carefully where they stand on this".
May, who is due to leave office in a week after a three-year premiership dominated by Brexit, condemned the "politics of division" and said "some are losing the ability to disagree without demeaning the views of others". "Ministers who walk the walk in the newspapers need to think very carefully and vote on ways to make proroguing Parliament more hard".
However, reports from Brussels said Brexit Secretary Stephen Barclay told the EU's chief negotiator, Michel Barnier, five times in a recent meeting that Mrs May's withdrawal deal is "dead".