WASHINGTON - President Barack Obama on Thursday defended America’s controversial drone attacks but announced new restraints on targeted killings, saying civilian casualties haunt him and that Washington is in the process of rebuilding relations with Pakistan.

In a major speech to military and political leaders at the National Defence University, Obama announced the most significant shift in US counter-terrorism since the fall of the World Trade Centre, saying he would restrict his own signature policy of ordering drone strikes around the world.

He said he has planned to wind down the drone strikes, which have stirred controversy at home and abroad. He imposed a higher standard on authorising such aerial attacks, shifted responsibility more from the CIA to the military and suggested the creation of a secret court that would have to sign off on strikes in the future.

President Obama said the US could not wage “a boundless global war on terror” but must face a new reality where threats come from regional jihadists and home-grown extremists. “America is at a crossroads” in the fight against terror... We must define the nature and scope of this struggle, or else it will define us… Our systematic effort to dismantle terrorist organisations must continue. But this war, like all wars, must end… That’s what history advises. That’s what our democracy demands.”

At the end of his hour-long speech, the president was repeatedly heckled by a woman who demanded the immediate release of detainees on hunger-strike at Guantanamo military prison. “Part of free speech is letting me speak,” Obama told the woman, who belonged to the social justice group “Code Pink”. During interruptions, the president however renewed his resolve to close down Guantanamo Bay and urged Congress to lift the restrictions on detainee transfers from Gitmo. “The voice of that woman is worth paying attention to... These are tough issues and the suggestion that we can gloss over them is wrong.”

Obama said Washington will continue to have the need to partner with foreign countries to counter terrorist threats as he acknowledged the sacrifices Pakistan has made. “Already, thousands of Pakistani soldiers have lost their lives in fighting terrorists,” Obama said.

“The cost of our relationship with Pakistan was immense... Our operation in Pakistan against Osama bin Laden cannot be the norm. The risks in that case were immense... the fact that we did not find ourselves confronted with civilian casualties, or embroiled in an extended firefight... also depended on some luck. And even then, the cost to our relationship with Pakistan – and the backlash among the Pakistani public over encroachment on their territory – was so severe that we are just now beginning to rebuild this important partnership.”

Another highly troublesome issue with Pakistan is the unending drone strikes. Under a new directive signed this week drone strikes would be limited only to those cases where the target represented a “continuing and imminent threat” to the US. Although insisting that the targeted killing programme was legal and effective, Obama said the US must also exercise “the discipline to constrain that power – or risk abusing it”. He insisted on “near certainty” that no civilians would be killed before authorising strikes, saying for him and his commanders the deaths of innocents would “haunt us as long as we live”.

Today, he said, the core of al-Qaeda in Afghanistan and Pakistan is on a path to defeat. “Their remaining operatives spend more time thinking about their own safety than plotting against us. They did not direct the attacks in Benghazi or Boston. They have not carried out a successful attack on our homeland since 9/11. Instead, what we’ve seen is the emergence of various al Qaeda affiliates. From Yemen to Iraq, from Somalia to North Africa, the threat today is more diffuse, with al-Qaeda’s affiliate in the Arabian Peninsula – AQAP –the most active in plotting against our homeland. While none of AQAP’s efforts approach the scale of 9/11 they have continued to plot acts of terror, like the attempt to blow up an airplane on Christmas Day in 2009.”

On the loss of civilian lives in US counterterrorism actions, he said “it is a hard fact that US strikes have resulted in civilian casualties, a risk that exists in all wars... For the families of those civilians, no words or legal construct can justify their loss. For me, and those in my chain of command, these deaths will haunt us as long as we live, just as we are haunted by the civilian casualties that have occurred through conventional fighting in Afghanistan and Iraq.”

He also stressed the utility of US assistance for foreign countries, calling it fundamental to American national security, and any sensible long-term strategy to battle extremism. “Foreign assistance is a tiny fraction of what we spend fighting wars that our assistance might ultimately prevent. For what we spent in a month in Iraq at the height of the war, we could be training security forces in Libya, maintaining peace agreements between Israel and its neighbours, feeding the hungry in Yemen, building schools in Pakistan, and creating reservoirs of goodwill that marginalise extremists.”

“For over the last decade, our nation has spent well over a trillion dollars on war, exploding our deficits and constraining our ability to nation build here at home. Our service-members and their families have sacrificed far more on our behalf. Nearly 7,000 Americans have made the ultimate sacrifice,” Obama said.

The White House released a policy document that Obama okayed ahead of his speech. The document lays out a rigorous process for reviewing and approving operations to capture or employ lethal force against terrorist targets outside the US and outside areas of active hostilities. The White House said Obama has decided to share, in this document, certain key elements of these standards and procedures with the American people so that they can make informed judgments and hold the Executive Branch accountable.

“This document provides information regarding counterterrorism policy standards and procedures that are either already in place or will be transitioned into place over time... We are continually working to refine, clarify, and strengthen our standards and processes for using force to keep the nation safe from the terrorist threat.”