Roxane Gay is worried.

Speaking to a packed Sydney Recital Hall on Friday night, the author of Difficult Woman, Bad Feminist and now Hunger, said she was deeply concerned about the election of Donald Trump, and what it meant for the United States.

Responding to an audience question, Gay described the Trump presidency as “catastrophic” for women and people of colour, particularly those who are working class or impoverished. “They are not on his radar, or the radar of his administration and when they are, they are targets.”

She added: “I’m very worried about the level of misogyny and the level of xenophobia and racism in America that we’re seeing.”

Continued resistance was the only option, she said: “We are going to have to continue to avoid this idea of resistance fatigue, which people are talking about, because every single day there is some new ridiculous revelation and some new sense of just how horrible he is. It’s really bad.”

But she did urge people to keep calm. “People keep saying The Handmaid’s Tale is right around the corner, and I think we need to calm down, and not catastrophise because that doesn’t do anything good.

“We are not living The Handmaid’s Tale, and a lot of what we are talking about, we’re talking about in the context of the developed world where obviously everybody is relatively OK, so let’s save the catastrophe for when it’s actually here.”

The US president has cast a long shadow over the Sydney writers’ festival this year with many authors unable to avoid talking about his impact. When speaking to author Thomas Friedman on Friday night, George Megalogenis reminded the audience of last year’s keynote speaker, Gloria Steinem, who, when asked about a potential Trump presidency, had said, “It will not happen.”

Also on Friday night, US authors George Saunders and Colson Whitehead and Guardian US data editor Mona Chalabi spoke with Slate editor-in-chief Julia Turner on a panel called “American carnage”, named for a phrase used in Trump’s bleak first speech as president.

The panel began with a statement from Turner that was greeted with applause: “There is really only one way for any upstanding American to open any international conversation about our new president Donald Trump: we are so, so sorry.”

Whitehead, who won the Pulitzer prize for his bestselling book The Underground Railroad, said Trump’s ascendence forced him to confront a truth he had been denying. “Obama lulled me into a sense that we had moved into a different place. A place where I didn’t have to fear as much for my kids growing up as my parents feared for me growing up. And I used to joke about this, ‘Oh, post-racial, that’s so stupid – we’re not post-racial!’ - and now I’d LOVE to make a joke about us being postracial.

“Everything that Donald represents is an essential part of who we in America are. It’s been there from the beginning and it will always be there,” he said.

He said he found himself struggling to empathise with people who voted for Trump: “You don’t have to be racist to vote for him, but it helps, and if you’re not racist you’re OK with having a racist sex abuser in the White House. So for me they are the two things: you’re either a racist or a total arsehole. So I have a hard time rising above myself to understand them.”

Chalabi, a Muslim immigrant who has been living in New York for three years, said the election and the travel ban had been “petrifying”.

“Some Americans tried to reassure me by pointing out the incompetence of the administration, that’s something I’ve heard again and again. But when you’re in a position of vulnerability that incompetence really isn’t reassuring because it means that things are really arbitrary and it’s very easy for you to slip between the cracks,” Chalabi said.

She told of an autocorrected text her mum sent her, which read: “Good Isis, see you at Heathrow”.

“I really don’t want that text on my phone,” she said. “And I’m saying this all from an enormously privileged position. I’ve been accepted into American society, I’m in a good social class, I have a good job, I speak English ... For me to be scared – I can’t imagine what other people are feeling.”

During Saturday night’s “Nasty Women” gala, the Wheeler Centre’s Sophie Black asked the US writers on stage what it was like to be on the other side of the world and asked, continually, to explain the Trump presidency. Brit Bennett, a writer and essayist whose first book The Mothers came out last year, said she embraces it.

“It’s such an insane time. The time difference – by the time I wake up here, the American news cycle is in full swing so I’m just like, ‘OK, what’s everyone panicking about today on Twitter?’”

But, Bennett said, there’s an upside to: “It’s been comforting to come here and talk to so many people who understand why we’re panicked and share that panic, to feel like we’re not just alone in our freaking out.”

Writer Nadja Spiegelman agreed: “In the United States everyone’s so tired of talking about it. I bring it up at a party and people just look really shellshocked and dead and they can’t speak about it anymore. And I can’t help it – it’s all I want to talk about, it obsesses me, it’s awful, it’s continually awful, but there’s no looking away from it. So it is kind of comforting coming here and having people as eager to talk about it as I am.”

Bennett said fiction could have a role to play in bridging the divide, by helping us empathise with and understand other people – so long as it was done in a thoughtful way. “I try to think about this in my own writing: whose voices I can identify with; whose experiences I centre,” she said. “I hope that this imagination and empathy can trickle into our politics. I don’t find a huge leap between creative empathy and political empathy: who you invite into your brain and heart when you read and write, and who you consider when you vote or protest or pass laws.

“So I hope that all of us that are rattled or maybe galvanised by this past election will build a way forward by embracing more complexity and more diversity, and always turning towards what we don’t understand.”

• Sydney writers’ festival continues until Sunday 28 May