In an exclusive interview with the Alan Brazil Sports Breakfast, Sullivan said: "We finished seventh last season largely down to him".
West Ham signed defender Jose Fonte from Southampton and midfielder Robert Snodgrass from Hull City this January to strengthen the team.
West Ham have now accepted a £25m offer from Marseille, up from £22m two weeks ago.
Payet, 29, joined West Ham in June 2015 from Marseille for $13.3 million, scoring 12 goals in his first season and named by the Professional Footballers' Association as player of the year.
Dimitri Payet's acrimonious divorce fromWest Ham ended late Sunday as the France global signed for former club Marseille in a 30 million euro (£25 million, $31.3m) deal.
Payet, who featured for France in last year's Euro 2016 championships where they lost to Portugal in the final, had been demoted to train with the Under-23 side as he was frozen out by Bilic and his team-mates after his refusal to play. There he thrived too, doing the same things in France's blue as he had in West Ham's colors.
Manager Slaven Bilic initially stated that the playmaker would not be sold but it quickly became apparent that there was no way back for Payet at the London Stadium.
Payet has gone from hero to zero at West Ham so quickly the club are offering to replace replica shirts sold with his name on the back.
France worldwide Payet, who arrived from Marseille in the summer of 2015, was a big hit in his first season with the London club, scoring 12 times in 38 appearances.
"We're going home", Payet tweeted alongside a photo of a small private plane with its door open at Marseille airport. "Marseille won't get there next season-they're now sixth in Ligue 1, 13 points away from third place and the Champions League qualifiers spot that comes with it-but the club does have a new, rich (?) owner, Frank McCourt (hence the "?"), who's promised to spend big to win. We were brilliant for him also.
The Frenchman eventually left the club after stating that he was unhappy.
"As much as we (the club's owners) didn't want him out, you can't go against the consensus of the team", Sullivan told the BBC, adding that Payet sacrificed his wages for January.