"You've also this sort of Bannon-bart influence in there, which I think is a snag on the president", Scaramucci told Stephanopoulos.
"I think he's also of the impression that there's hatred on all sides, but I disagree with it", he said.
Scaramucci was sacked only 10 days into his job as communications director after the New Yorker published an obscenity-laced interview he had with a reporter.
Later Sunday, Trump's national security adviser H.R. McMaster - who has been on the receiving end of criticism from Breitbart and Bannon allies - repeatedly deflected questions on NBC's Meet the Pressabout whether he could work with Bannon in the White House.
"It's actually terrorism and whether it's domestic or global terrorism, with the moral authority of the presidency, you have to call that stuff out", he said.
In his first interview since being fired from the Trump administration after less than two weeks on the job, Scaramucci looked on, sometimes ruefully nodding and smiling, as host George Stephanopoulos played some of the more colorful parts of the vulgar interview that led to his dismissal.
He said the "toleration of [white supremacy] by Steve Bannon is inexcusable".
Anthony Scaramucci, who was sacked after a tempestuous 11 days as the White House communications director, likened his job to a character in the movie "Pulp Fiction".
"The whole thing is nonsensical".
He also called on Trump to distance himself from top adviser Steve Bannon, the divisive former head of Breitbart News who Scaramucci argued was pulling Trump too far to the right and was "not serving the president's interest". "I think the president knows what he's going to do".
When discussing the influences that he thinks are impacting Trump, Scaramucci urged the president to take a more mainstream approach in order to sell his agenda.
Scaramucci said during the interview that he accepts that what he did was wrong, but emphasized that people need to move on and focus on the president's agenda. If he does that, he'll have a successful legislative agenda that he can execute.