BOSTON –- Mitt Romney's campaign manager admitted regret that the Republican challenger took such a hardline stance on immigration during the GOP primary, one of the first clear acknowledgements by the Romney campaign that its candidate hurt himself among Latino voters.

Matt Rhoades, who oversaw Romney's campaign, gathered with other top decision makers from both Romney's and President Barack Obama's campaigns at Harvard University's Institute of Politics Thursday for the "Campaign Managers Conference," which the school has hosted every four years since 1972. The session was on the record, but embargoed until an audio recording was posted online Monday.

During a discussion of the primary, Rhoades and top Romney strategist Stuart Stevens were asked whether they regretted attacking former GOP candidate Texas Gov. Rick Perry on the immigration issue in the summer and fall of 2011.

Stevens, 59, shook his head. But Rhoades, 37, answered the question.

"I regret that ..." Rhoades said, before stopping himself, and phrasing his answer differently.

Rhoades went on to describe how the Romney campaign may have regarded Perry as a mortal threat for too long, leading it to engage him beyond the point when it was necessary, and setting itself up for a hard-right turn on immigration.

"If you look through the unwinding of the Perry campaign, a lot of people put a focus on that one infamous debate moment," Rhoades said, referring to Perry's epic "oops" moment at an early November debate in Detroit, when Perry could not remember the three federal agencies he planned to eliminate.

"But it was the earliest debates, the first and second debates," Rhoades said. "And by the third debate, and this was well before the other moment, I think Governor Perry was badly hurt."

Rhoades reflected on how Romney had inflicted damage on Perry in early September with attacks on his comments that Social Security was a "Ponzi scheme" and his plan to turn the plan over to state governments.

"In retrospect, I believe we probably could have just beaten Governor Perry with the Social Security hit," Rhoades said.

Perry's support in Texas of giving in-state college tuition rates to children of undocumented immigrants did not become an issue until a Sept. 22 debate, when former Pennsylvania Sen. Rick Santorum attacked him for his stance, prompting Perry to say that those who disagreed with him "don't have a heart." Romney pounced on Perry's comment, and from there, he ran hard to the right of Perry on immigration. Any chance that Perry might recover from his terrible performances in most of the debates was extinguished in Iowa, ahead of the state's Jan. 3 caucuses, by the immigration issue.

A hardline stance on immigration was a big part of Romney's primary race platform from late September on, even after Perry faded. A few days before the Iowa caucuses, as Romney was locked in a battle with Santorum for the state, he vowed to veto the Dream Act, a plan to give some children of undocumented immigrants a path to citizenship.

And perhaps most harmful to his relationship with Latinos, Romney said he supported creating an environment where undocumented immigrants "self-deported." Prominent Republicans such as former Florida Gov. Jeb Bush criticized Romney for this stance.

But the damage was done. Romney lost Latino voters by a huge margin in the general election last month, getting only 27 percent of the nation's biggest and fastest-growing minority group. In 2004, President George W. Bush got between 40 and 44 percent of the Latino vote, and in 2008, Sen. John McCain (R-Ariz) did almost as badly as Romney, with only 31 percent.

Even before the election this year, Republican operatives such as Ed Gillespie -– who went on to work in a senior role on the Romney campaign -– were warning that if the GOP did not soften its sometimes "anti-immigrant" rhetoric and become a party more welcoming to Latinos, it would be harm the party's long-term future.

In the wake of a second consecutive election where the Republican presidential candidate performed woefully among the key voting bloc, many in the party are resolving that short-term political considerations can no longer determine the party's course on the issue. One Republican consultant who worked for Romney told The Huffington Post after the election that he "kept a lot of conservatives away from [Perry] on immigration because I had to."

He added: "I won't ever do it again."

David Axelrod, Obama's chief strategist on his reelection campaign, said at the Harvard conference that Romney's positioning on immigration was indicative of a broader pattern in the Republican's campaign.

"Here's my observation of the Romney campaign. It seems to me that they always were doing what they needed to get through the next thing, on the theory that just being on the ballot against Barack Obama, as vulnerable as he was, was enough," Axelrod said. "And so, if you had to run to the right of Perry on immigration, you run to the right of Perry on immigration. If you have to run to the right of Santorum on social issues, you run to the right of Santorum."

Rhoades, however, said it was hard to think about the general election during the primary.

"When you're running for the nomination, you've got to win the nomination. If you're looking beyond securing the nomination too much, you are jeopardizing your chances of winning the nomination," he said.

Also on HuffPost:

Loading Slideshow...
  • The Naturalization Act of 1790

    The Naturalization Act of 1790 was our country's first set of laws dealing with citizenship. Applicants had to be "<a href="http://rs6.loc.gov/cgi-bin/ampage?collId=llsl&fileName=001/llsl001.db&recNum=226 " target="_hplink">a free white person</a>" of "good moral character." This excluded indentured servants and slaves. Good moral character was substantiated by establishing residence for at least one year in the state from where he was applying, and at least two years of residence in the country. The Naturalization Act of 1795 would extend that requirement to five years, and is still standard today.

  • The Fourteenth Amendment, 1868

    A Reconstruction Amendment that was added to the U.S. Constitution following the Civil War, the Citizenship Clause of the 14th Amendment establishes for the first time that children born on U.S. soil would be conferred U.S. citizenship regardless of their parent's citizenship status, race, or place of birth. Last year, Rep. Steve King (R-IA) introduced the <a href="http://www.govtrack.us/congress/bills/112/hr140 " target="_hplink">Birthright Citizenship Act of 2011</a> to Congress, and challenged this. The bill would require that at least one parent be a U.S. citizen or permanent resident for a child to be granted citizenship. According to the <a href="http://www.opencongress.org/bill/112-h140/text " target="_hplink">bill's text</a>, the Birthright Citizenship Act of 2011 would amend the Immigration and Nationality Act of 1952, and "clarify those classes of individuals born in the United States who are nationals and citizens of the United States at birth." Prior to this, Rep. Nathan Deal (R-GA) <a href="http://www.huffingtonpost.com/2009/05/26/nathan-deal-georgia-lawma_n_207485.html " target="_hplink">introduced</a> a similar <a href="http://www.opencongress.org/bill/111-h1868/show" target="_hplink">bill</a> in 2009.

  • The Naturalization Act of 1870

    The Naturalization Act of 1870<a href="http://thepoliticsofimmigration.org/pages/chronology.htm " target="_hplink"> explicitly extended</a> naturalization laws to "aliens of African nativity and persons of African descent." This meant that for the first time, African-American children would be conferred citizenship upon birth. Asian immigrants and other people of color are excluded per the Naturalization Acts of 1790 and 1795.

  • The Page Act of 1875

    Named after Republican Representative Horace F. Page, this is the first U.S. federal immigration law to explicitly prohibit the immigration of a particular group: persons of Asian descent. Primarily meant to limit Chinese immigrant labor and prostitution, the Page Act prohibited the immigration of: (1) contracted labor from "China, Japan, or any Oriental country" that was not "free and voluntary," (2) Chinese prostitution and (3) criminals and women who would engage in prostitution. Ultimately, the <a href="http://www.uchastings.edu/racism-race/pageact.html " target="_hplink">Page Act</a> severely <a href="http://immigration-online.org/228-page-act-united-states-1875.html " target="_hplink">restricted</a> the immigration of Asian women. Only 136 of the the nearly 40,000 Chinese immigrants who arrived in the months before the bill's enforcement were women. And, it would pave the way for the Chinese Exclusion Act. In this picture, Michael Lin, chair of the 1882 Project, a coalition of rights groups seeking a statement of regret over that year's Chinese Exclusion Act, speaks on May 26, 2011 in Washington, DC, at the US House of Representatives in front of a reproduction of a 19th-century sign that aimed at rousing up sentiment against Chinese Americans. Lawmakers introduced a bill that would offer an official statement of regret for the act, which banned further immigration of Chinese to the United States and ended citizenship rights for ethnic Chinese. (AFP PHOTO/SHAUN TANDON).

  • The Chinese Exclusion Act, 1882

    Signed by President Chester A. Arthur, the <a href="http://www.pbs.org/weta/thewest/resources/archives/seven/chinxact.htm " target="_hplink">Chinese Exclusion Act</a> was the first federal immigration law to prohibit immigration on the basis of race. The bill barred all Chinese laborers, skilled and unskilled, from immigrating to the U.S. for ten years. It was made permanent by 1903, and was not lifted until the 1943 Magnuson Act. The 1898 Supreme Court <a href="http://ocp.hul.harvard.edu/immigration/exclusion.html " target="_hplink">decision</a> in <em>United States v. Wong Kim Ark</em> finally extended naturalization laws to persons of Chinese descent by ruling that anyone born in the United States was indeed a U.S. citizen. This editorial cartoon from 1882 shows a Chinese man being excluded from entry to the "Golden Gate of Liberty." The sign next to the iron door reads, "Notice--Communist, Nihilist, Socialist, Fenian & Hoodlum welcome. But no admittance to Chinamen." At the bottom, the caption reads, "THE ONLY ONE BARRED OUT. Enlightened American Statesman--'We must draw the line <em>somewhere</em>, you know.'" (Image Source: Frank Leslie's illustrated newspaper, vol. 54 (1882 April 1), p. 96. [Public domain], via <a href="http://commons.wikimedia.org/wiki/File:The_only_one_barred_out_cph.3b48680.jpg" target="_hplink">Wikimedia Commons</a>).

  • The Naturalization Act of 1906

    The Naturalization Act of 1906 further <a href="http://www.understandingrace.org/history/gov/eastern_southern_immigration.html" target="_hplink">defined</a> the naturalization process: the ability to speak English was made a <a href="http://www.enotes.com/topic/Naturalization_Act_of_1906" target="_hplink">requisite</a> for immigrants to adjust their status.

  • The Immigration Act of 1924

    U.S. President Coolidge signed this U.S. federal <a href="http://history.state.gov/milestones/1921-1936/ImmigrationAct " target="_hplink">bill</a> into law. It capped the number of immigrants who could be admitted entry to the U.S. and barred immigration of persons who were not eligible for naturalization. And, as the Naturalization Act of 1790 required, an immigrant had to be white in order to naturalize. The quotas varied by country. Image Source: Flickr Creative Commons, <a href="http://www.flickr.com/photos/nycmarines/6306315902/" target="_hplink">NYCMarines</a>.

  • The Immigration and Nationality Act of 1952 (The McCarran-Walter Act)

    The <a href="https://docs.google.com/viewer?a=v&q=cache:zwaVG82lZisJ:www-rohan.sdsu.edu/dept/polsciwb/brianl/docs/1952McCarranWaltersAct.pdf+&hl=en&gl=us&pid=bl&srcid=ADGEESjEwx76FIBTixZAfyncZz-1CSuSeciv5qB6vvWTrUfW58XRpXq8zkpnI57XSuuG5Bu-WSySGbEhxYvZxP7y6qDQuOsDhgDa6qUqUaJ8F4imTzKJsVtppHc_-eew2dK6vGhoIUZs&sig=AHIEtbTNQ5GFiNMVS-xyThq8VVSj_gG9KA " target="_hplink">McCarran-Walter Act</a> kept up the controversial Immigration Act of 1924, but <a href="http://history.state.gov/milestones/1945-1952/ImmigrationAct" target="_hplink">formally</a> ended Asian exclusion.

  • Immigration and Nationality Act of 1965

    When President Lyndon Johnson signed the Immigration and Nationality Act of 1965, it <a href="http://library.uwb.edu/guides/USimmigration/1965_immigration_and_nationality_act.html" target="_hplink">abolished</a> the quota system that favored immigration from Europe and limited immigration from Asia and South America.

  • Illegal Immigration Reform and Immigrant Responsibility Act of 1996

    The 1996 <a href="http://www.uscis.gov/ilink/docView/PUBLAW/HTML/PUBLAW/0-0-0-10948.html " target="_hplink">Illegal Immigration Reform and Immigrant Responsibility Act</a> (IIRIRA) is a piece of legislation that <a href="http://library.uwb.edu/guides/usimmigration/1996_illegal_immigration_reform_and_immigrant_responsibility_act.html " target="_hplink">defined</a> an array of issues to do with legal and illegal immigration -- from outlining how border patrol agents should administer visa processing, to the minutiae of how to handle deportation proceedings -- IIRIRA established enforcement and patrolling practices.