And on Monday, a close friend of Trump's said the president is considering firing Mueller, a move that echoes Richard Nixon ordering the termination of the special prosecutor during the Watergate scandal.
"I have never been directed to do anything I believe to be illegal, immoral, unethical or inappropriate", Rogers said.
It is not clear whether the White House could use executive privilege to try to block them from speaking to the investigators.
Mueller was appointed by Deputy Attorney General Rod Rosenstein.
Mueller is leading the investigation into Russia's meddling in the 2016 election and potential ties between Moscow and Trump's presidential campaign.
Mr Trump claimed he had been personally vindicated by Mr Comey's testimony last week confirming that he was not under investigation himself for links to Russian Federation. "He comes from a law firm that represents members of the Trump family", Ruddy said.
After news of Ruddy's interview surfaced, Rep. Adam Schiff, the top Democrat on the House intelligence committee, tweeted: "If President fired Bob Mueller, Congress would immediately re-establish independent counsel and appoint Bob Mueller".
"Time to rethink", he tweeted on Monday, citing Mr Mueller's hiring decisions and Mr Comey's admission that he instructed a friend to share with reporters notes he had taken of his private conversations with Mr Trump in order to force the appointment of special counsel.
Trump's allies have begun raising questions about special counsel Robert Mueller's impartiality - he's a former FBI director who has worked with fired FBI Director James Comey - and floating the idea that Trump might replace him.
"I think he's considering perhaps terminating the special counsel", Ruddy said.
After the meeting with Mueller, Warner was asked by CNN's Manu Raju whether the committee is looking into obstruction of justice or if that was part of the special counsel investigation.
"We're not going to let the President choose who conducts this investigation", he said.
The letter said the investigation will also probe Comey's testimony that Loretta Lynch, as President Barack Obama's attorney general, had directed him to describe an FBI probe into Hillary Clinton's email practices as merely a "matter" and to avoid calling it an investigation.
Peter Carr, a spokesman for Mueller, declined to comment on Ruddy's remarks.
The leaders of the Senate Intelligence committee said in a statement issued Wednesday that they "look forward to future engagements" with Robert Mueller. "Don't waste our time".