The charge comes from the adviser, Richard Clarke, in an exclusive interview on 60 Minutes.
The administration maintains that it cannot find any evidence that the conversation about an Iraq-9/11 tie-in ever took place.
Clarke also tells CBS News Correspondent Lesley Stahl that White House officials were tepid in their response when he urged them months before Sept. 11 to meet to discuss what he saw as a severe threat from al Qaeda.
"Frankly," he said, "I find it outrageous that the president is running for re-election on the grounds that he's done such great things about terrorism. He ignored it. He ignored terrorism for months, when maybe we could have done something to stop 9/11. Maybe. We'll never know."
Clarke went on to say, "I think he's done a terrible job on the war against terrorism."
The No. 2 man on the president's National Security Council, Stephen Hadley, vehemently disagrees. He says Mr. Bush has taken the fight to the terrorists, and is making the U.S. homeland safer.
Clarke says that as early as the day after the attacks, Secretary of Defense Donald Rumsfeld was pushing for retaliatory strikes on Iraq, even though al Qaeda was based in Afghanistan.
Clarke suggests the idea took him so aback, he initally thought Rumsfeld was joking.
Clarke is due to testify this week before the special panel probing whether the attacks were preventable.
His allegations are also made in a book, "Against All Enemies," by Free Press, a subsidiary of Simon & Schuster. Both CBSNews.com and Simon & Schuster are units of Viacom.
Clarke helped shape U.S. policy on terrorism under President Reagan and the first President Bush. He was held over by President Clinton to be his terrorism czar, then held over again by the current President Bush.
In the 60 Minutes interview and the book, Clarke tells what happened behind the scenes at the White House before, during and after Sept. 11.
When the terrorists struck, it was thought the White House would be the next target, so it was evacuated. Clarke was one of only a handful of people who stayed behind. He ran the government's response to the attacks from the Situation Room in the West Wing.
"I kept thinking of the words from 'Apocalypse Now,' the whispered words of Marlon Brando, when he thought about Vietnam. 'The horror. The horror.' Because we knew what was going on in New York. We knew about the bodies flying out of the windows. People falling through the air. We knew that Osama bin Laden had succeeded in bringing horror to the streets of America," he tells Stahl.
After the president returned to the White House on Sept. 11, he and his top advisers, including Clarke, began holding meetings about how to respond and retaliate. As Clarke writes in his book, he expected the administration to focus its military response on Osama bin Laden and al Qaeda. He says he was surprised that the talk quickly turned to Iraq.
"Rumsfeld was saying that we needed to bomb Iraq," Clarke said to Stahl. "And we all said ... no, no. Al-Qaeda is in Afghanistan. We need to bomb Afghanistan. And Rumsfeld said there aren't any good targets in Afghanistan. And there are lots of good targets in Iraq. I said, 'Well, there are lots of good targets in lots of places, but Iraq had nothing to do with it.
"Initially, I thought when he said, 'There aren't enough targets in-- in Afghanistan,' I thought he was joking.
"I think they wanted to believe that there was a connection, but the CIA was sitting there, the FBI was sitting there, I was sitting there saying we've looked at this issue for years. For years we've looked and there's just no connection."
Clarke says he and CIA Director George Tenet told that to Rumsfeld, Secretary of State Colin Powell, and Attorney General John Ashcroft.
Clarke then tells Stahl of being pressured by Mr. Bush.
"The president dragged me into a room with a couple of other people, shut the door, and said, 'I want you to find whether Iraq did this.' Now he never said, 'Make it up.' But the entire conversation left me in absolutely no doubt that George Bush wanted me to come back with a report that said Iraq did this.
"I said, 'Mr. President. We've done this before. We have been looking at this. We looked at it with an open mind. There's no connection.'
"He came back at me and said, "Iraq! Saddam! Find out if there's a connection.' And in a very intimidating way. I mean that we should come back with that answer. We wrote a report."
Clarke continued, "It was a serious look. We got together all the FBI experts, all the CIA experts. We wrote the report. We sent the report out to CIA and found FBI and said, 'Will you sign this report?' They all cleared the report. And we sent it up to the president and it got bounced by the National Security Advisor or Deputy. It got bounced and sent back saying, 'Wrong answer. ... Do it again.'
"I have no idea, to this day, if the president saw it, because after we did it again, it came to the same conclusion. And frankly, I don't think the people around the president show him memos like that. I don't think he sees memos that he doesn't-- wouldn't like the answer."
Clarke was the president's chief adviser on terrorism, yet it wasn't until Sept. 11 that he ever got to brief Mr. Bush on the subject. Clarke says that prior to Sept. 11, the administration didn't take the threat seriously.
"We had a terrorist organization that was going after us! Al Qaeda. That should have been the first item on the agenda. And it was pushed back and back and back for months.
"There's a lot of blame to go around, and I probably deserve some blame, too. But on January 24th, 2001, I wrote a memo to Condoleezza Rice asking for, urgently -- underlined urgently -- a Cabinet-level meeting to deal with the impending al Qaeda attack. And that urgent memo-- wasn't acted on.
"I blame the entire Bush leadership for continuing to work on Cold War issues when they back in power in 2001. It was as though they were preserved in amber from when they left office eight years earlier. They came back. They wanted to work on the same issues right away: Iraq, Star Wars. Not new issues, the new threats that had developed over the preceding eight years."
Clarke finally got his meeting about al Qaeda in April, three months after his urgent request. But it wasn't with the president or cabinet. It was with the second-in-command in each relevant department.
For the Pentagon, it was Paul Wolfowitz.
Clarke relates, "I began saying, 'We have to deal with bin Laden; we have to deal with al Qaeda.' Paul Wolfowitz, the Deputy Secretary of Defense, said, 'No, no, no. We don't have to deal with al Qaeda. Why are we talking about that little guy? We have to talk about Iraqi terrorism against the United States.'
"And I said, 'Paul, there hasn't been any Iraqi terrorism against the United States in eight years!' And I turned to the deputy director of the CIA and said, 'Isn't that right?' And he said, 'Yeah, that's right. There is no Iraqi terrorism against the United States."
Clarke went on to add, "There's absolutely no evidence that Iraq was supporting al Qaeda, ever."
When Stahl pointed out that some administration officials say it's still an open issue, Clarke responded, "Well, they'll say that until hell freezes over."
By June 2001, there still hadn't been a Cabinet-level meeting on terrorism, even though U.S. intelligence was picking up an unprecedented level of ominous chatter.
The CIA director warned the White House, Clarke points out. "George Tenet was saying to the White House, saying to the president - because he briefed him every morning - a major al Qaeda attack is going to happen against the United States somewhere in the world in the weeks and months ahead. He said that in June, July, August."
Clarke says the last time the CIA had picked up a similar level of chatter was in December, 1999, when Clarke was the terrorism czar in the Clinton White House.
Clarke says Mr. Clinton ordered his Cabinet to go to battle stations-- meaning, they went on high alert, holding meetings nearly every day.
That, Clarke says, helped thwart a major attack on Los Angeles International Airport, when an al Qaeda operative was stopped at the border with Canada, driving a car full of explosives.
Clarke harshly criticizes President Bush for not going to battle stations when the CIA warned him of a comparable threat in the months before Sept. 11: "He never thought it was important enough for him to hold a meeting on the subject, or for him to order his National Security Adviser to hold a Cabinet-level meeting on the subject."
Finally, says Clarke, "The cabinet meeting I asked for right after the inauguration took place-- one week prior to 9/11."
In that meeting, Clarke proposed a plan to bomb al Qaeda's sanctuary in Afghanistan, and to kill bin Laden.
The president's new campaign ads highlight his handling of Sept. 11 -- which has become the centerpiece of his bid for re-election.
"You are writing this book in the middle of this campaign," Stahl tells Clarke. "The timing, I'm sure, you will be questioned about and criticized for. Why are you doing it now?"
"Well, I'm sure I'll be criticized for lots of things," says Clarke. "And I'm sure they'll launch their dogs on me."
Does a person who works for the White House owe the president his loyalty?
"Yes ... Up to a point. When the president starts doing things that risk American lives, then loyalty to him has to be put aside," says Clarke. "I think the way he has responded to al Qaeda, both before 9/11 by doing nothing, and by what he's done after 9/11 has made us less safe. Absolutely."
Hadley staunchly defended the president to Stahl: "The president heard those warnings. The president met daily with ... George Tenet and his staff. They kept him fully informed and at one point the president became somewhat impatient with us and said, 'I'm tired of swatting flies. Where's my new strategy to eliminate al Qaeda?'"
Hadley says that, contrary to Clarke's assertion, Mr. Bush didn't ignore the ominous intelligence chatter in the summer of 2001.
"All the chatter was of an attack, a potential al Qaeda attack overseas. But interestingly enough, the president got concerned about whether there was the possibility of an attack on the homeland. He asked the intelligence community: 'Look hard. See if we're missing something about a threat to the homeland.'
"And at that point various alerts went out from the Federal Aviation Administration to the FBI saying the intelligence suggests a threat overseas. We don't want to be caught unprepared. We don't want to rule out the possibility of a threat to the homeland. And therefore preparatory steps need to be made. So the president put us on battle stations."
Hadley asserts Clarke is "just wrong" in saying the administration didn't go to battle stations.
As for the alleged pressure from Mr. Bush to find an Iraq-9/11 link, Hadley says, "We cannot find evidence that this conversation between Mr. Clarke and the president ever occurred."
When told by Stahl that 60 Minutes has two sources who tell us independently of Clarke that the encounter happened, including "an actual witness," Hadley responded, "Look, I stand on what I said."
Hadley maintained, "Iraq, as the president has said, is at the center of the war on terror. We have narrowed the ground available to al Qaeda and to the terrorists. Their sanctuary in Afghanistan is gone; their sanctuary in Iraq is gone. Saudi Arabia and Pakistan are now allies on the war on terror. So Iraq has contributed in that way in narrowing the sanctuaries available to terrorists."
Does Clarke think that Iraq, the Middle East and the world is better off with Saddam Hussein out of power?
"I think the world would be better off if a number of leaders around the world were out of power. The question is what price should the United States pay," says Clarke. "The price we paid was very, very high, and we're still paying that price for doing it."
"Osama bin Laden had been saying for years, 'America wants to invade an Arab country and occupy it, an oil-rich Arab country. He had been saying this. This is part of his propaganda," adds Clarke.
"So what did we do after 9/11? We invade an oil-rich and occupy an oil-rich Arab country which was doing nothing to threaten us. In other words, we stepped right into bin Laden's propaganda. And the result of that is that al Qaeda and organizations like it, offshoots of it, second-generation al Qaeda have been greatly strengthened."
When Clarke worked for Mr. Clinton, he was known as the terrorism czar. When Mr. Bush came into office, though remaining at the White House, Clarke was stripped of his Cabinet-level rank.
Stahl said to Clarke, "They demoted you. Aren't you open to charges that this is all sour grapes, because they demoted you and reduced your leverage, your power in the White House?"
Clarke's answer: "Frankly, if I had been so upset that the National Coordinator for Counter-terrorism had been downgraded from a Cabinet level position to a staff level position, if that had bothered me enough, I would have quit. I didn't quit."
Until two years later, after 30 years in government service.
A senior White House official told 60 Minutes he thinks the Clarke book is an audition for a job in the Kerry campaign.
"I'm an independent. I'm not working for the Kerry campaign," says Clarke. "I have worked for Ronald Reagan. I have worked for George Bush the first, I have worked for George Bush the second. I'm not participating in this campaign, but I am putting facts out that I think people ought to know."
60 Minutes received a note from the Pentagon saying: "Any suggestion that the president did anything other than act aggressively, quickly and effectively to address the al Qaeda and Taliban threat in Afghanistan is absurd."
