He said that it was very important to remember that "there are thousands of people working for Oxfam in some of the most hard places. and we must not forget that Oxfam is one of Britain's most brilliant charities".
Oxfam said it was "dismayed by what happened" and would fully cooperate with authorities.
"The reports of what is unacceptable behaviour by senior aid workers in Haiti are truly shocking", a spokeswoman for Theresa May said.
Former Oxfam employees in Chad claimed staff held sex parties with prostitutes.
Mordaunt spoke Sunday - three days after a Times of London investigation accused Oxfam's then-director in Haiti, along with other workers, of running an illegal makeshift brothel after a 2010 quake devastated the country.
Global development secretary Penny Mordaunt said it was "despicable" that sexual exploitation and abuse existed in the aid sector and threatened to cut funding to charities if they fail to comply with authorities over safeguarding issues.
There had been other incidents since 2011, he said, but "nothing on the scale of this severity".
Caroline Thomson, chair of Oxfam's trustees, announced a package of measures to show the organisation was committed to changing.
Oxfam Chief Executive Mark Goldring said Saturday it receives less than 10 percent of its funding from DFID and hoped to continue working with the department while rebuilding trust with the public.
He added: 'Everybody - the 25,000 staff and volunteers - are compromised by this, the hundreds of thousands of people who support Oxfam every month are compromised by this, and to everybody I apologise.
Penny Mordaunt condemned the behaviour of some Oxfam staff members as a "complete betrayal", as she warned the charity the "scandal" had put its relationship with the Government at risk.
"I have so much respect for Oxfam, they do great work, but this is a sector-wide problem". The charity allowed staff members to resign from the organisation and seek new aid jobs elsewhere.
She told the BBC's Andrew Marr Show: I am affording them the opportunity to tell me in person what they did after these events and I'm going to be looking to see if they are displaying the moral leadership that I think they need to now.
"If the moral leadership at the top of the organization isn't there, then we can't have you as a partner", she said.
Mordaunt also said she suspected some people were trying to join charities in order to carry out "predatory activities".
Marr said: "That was a lie, wasn't it?"
More reports of misconduct have emerged since the original report in the Times.
Ms Mordaunt replied: "Well, quite".
She added that Oxfam had done "absolutely the wrong thing" by failing to inform authorities about the full details of the allegations.
She added she was writing to all United Kingdom charities which receive United Kingdom aid asking them to spell out the steps they are taking to ensure safeguarding policies are in place and working properly.