In July 2016 the bigoted troll Milo Yiannopoulos, a British darling of the American far right, was banned from Twitter after encouraging a torrent of racist abuse at Leslie Jones, a black actor who starred in the remake of Ghostbusters. In one example he branded her “barely literate”. A few months later it emerged that Threshold Editions, an imprint of Simon & Schuster, had given Yiannopoulos, 33, who calls feminism “cancer” and Donald Trump “Daddy”, a $255,000 book deal. “I met with top execs … earlier in the year and spent half an hour trying to shock them with lewd jokes and outrageous opinions,” he said. “I thought they were going to have me escorted from the building – but instead they offered me a wheelbarrow full of money.”

A few months after that, he suggested that sex between 13-year-old boys and adult men and women could “happen perfectly consensually”. This was apparently one “outrageous opinion” too far. Simon & Schuster cancelled the book. Yiannopoulos sued.

In the subsequent court submissions we see the critical notes of his editor, Mitchell Ivers. Yiannopoulos argues: “Given my penchant for black denizens of the dark continent I can’t be accused of being racist … I’m the left’s worst nightmare.” Ivers replies: “Having sex with black people does not prove someone is [not] racist.” Elsewhere the editor demands: “Delete irrelevant and superfluous ethnic joke.”

In a section on feminism, Ivers says: “Don’t start chapter with accusation that feminists = fat.” Tellingly, he adds: “It destroys any seriousness of purpose.”

When the notes were made public, some liberals hailed them as evidence of Yiannopoulos’s shallow thinking. But they actually lay bare a far more sinister process. Simon & Schuster knew who he was when it signed him. Ivers’ job was to get him into shape, to coach him in how to make his racism and misogyny palatable. What we see in those notes is the strenuous, and ultimately doomed, effort to lend Yiannopoulos’s bigotry gravity; to extract from the dung heap of his hateful worldview “the seriousness of [his] purpose”; to locate the boundaries of acceptable prejudice so that those borders can be more effectively breached. He was not just an editor but an enabler.

This is the electoral challenge of the extreme right in the west: to find a plausible balance between how racist it actually is, in its policies, and how racist it can appear to be in its pronouncements. Its raison d’etre is to promote and project a mythical sense of national and racial purity; its conundrum is how to simultaneously attract racists and xenophobes to that project while denouncing racism and xenophobia. In short, to deny any connection between its agenda and the intended effect.

Milo Yiannopoulos.
Pinterest
Milo Yiannopoulos. Photograph: Lisa Maree Williams/Getty Images

This contradiction has been most glaringly apparent in the travails of the UK Independence party. Its most recent crisis was triggered by the disclosure of racist texts by Jo Marney, the girlfriend of the party leader, Henry Bolton. Among other things, when asked, “What’s wrong with black people?” Marney responded “ugly”, and when subsequently accused of being racist she remarked “lol so what”. Particularly upset by Prince Harry’s black fiancee, Meghan Markle, she wrote: “This is Britain not Africa,” and that Markle would “taint the royal seed”, and she worried that black people were “pushing their way to the top slowly, slowly. Next will be a Muslim PM … And a black king.”

Bolton split up with Marney (for now) and suspended her from the party. Nonetheless, senior members of Ukip, including Bolton’s deputy, have now resigned en masse, calling for him to step down.

It’s not difficult to see where Marney got these views from. Ukip ran its election campaign on banning the burqa and enforcing genital checks on girls from groups (by which it means Muslim) at risk of suffering female genital mutilation. Its former leader Nigel Farage once said: “Any normal and fair-minded person would have a perfect right to be concerned if a group of Romanian people suddenly moved in next door.” One of its MEPs said Britain should stop sending aid to “bongo bongo land”.

During the Brexit referendum, Leave.EU, the organisation with which Ukip associated, targeted its adverts on Facebook towards, among others, members of the far-right British National party and Britain First.

So Marney is no outlier. With the Brexit referendum out of the way it’s really just the racism, Islamophobia and xenophobia keeping the party together at 3% in the polls. It’s a wonder they haven’t made Bolton leader for life and put Marney on the frontbench.

Marine Le Pen with her father, Jean-Marie Le Pen, in 2014.
Pinterest
Marine Le Pen with her father, Jean-Marie Le Pen, in 2014. Photograph: Robert Pratta/Reuters

This is not a problem confined to Ukip or Britain. Within hours of Alternative für Deutschland (AfD) riding a wave of anti-immigrant sentiment into the German Bundestag last year, one of its leading members had left the parliamentary group and stormed out of its victory news conference. Frauke Petry, who considers herself a “moderate”, had previously failed to have a senior official expelled for denying Adolf Hitler was “absolutely evil” and branding Berlin’s Holocaust memorial a “monument of shame”.

In France, the Front National leader, Marine Le Pen, expelled her own father from the party after he repeated his view that gas chambers were a mere “detail of history” and called on France to unite with Russia to save the “white world”. Almost every day the US president says something egregious about one minority or another only to have the Trump whisperers in the Republican party and Fox News explain why, whatever else it was, it definitely wasn’t racist.

There is method to this madness. Petry said that the AfD, under hardliners, lacked a credible plan to govern; Marine Le Pen called her father’s comments political suicide. In other words, they wanted to win. And while racism and xenophobia were central to their meaning, they needed them to be coded in their message. To achieve this they must relegate racism from an issue of power and discrimination to a matter of politeness and decorum.

Meanwhile, hate crimes against Muslims and refugees are on the rise in Germany; and against African Americans, Jews and Muslims in the US. Hate crimes in the UK leapt 29% last year.

These parties often look ridiculous. Bolton is Ukip’s sixth leader in 15 months (two of those have been Farage). Trump is a buffoon. This week an AfD executive member in the state of Brandenburg converted to Islam. The party’s website states: “Islam does not belong in Germany.”

The trouble is they are also incredibly effective. They won the Brexit referendum; Trump is in the Oval Office; the Front National made the presidential runoffs again; the AfD became the third largest party in Germany.

“We are what we pretend to be,” said the late novelist Kurt Vonnegut. “So we must be careful about what we pretend to be.” For too long we have pretended we are tolerant societies in which racism is not a system of oppression but the marginal obsession of the uncouth. In reality we have simply become more sophisticated about our prejudices. We have plenty of racism, but apparently very few racists.

Gary Younge writes a weekly column for the Guardian