Pinchas Gutter gazes at you with arresting blue eyes as he describes how his family was destroyed in the Holocaust. “My parents, my mother, my father and my sister – my twin sister – they were murdered the same day, when we arrived at the death camp,” he says. Now at 83, Gutter has only a fleeting memory of his sister. “All I remember is the braid,” he says. “She had long brown hair that my mother used to plait.”

Survivors such as Gutter are often invited to speak at museums and schools, as their personal stories have become critical to educating younger generations. Witness testimony is an inescapable, “life-breathing force” that “we cannot deny”, wrote Professor Leo Lieberman, of the Richard Stockton College in New Jersey. Soon, though, there will be no Holocaust survivors left.

“This is a really critical time,” says Dr David Traum at the University of Southern California. “Children today will be the last generation to hear from living survivors.”

But a new project means Gutter’s story has escaped mortality, and he will be telling audiences about his life for decades, perhaps even centuries, to come.

Gutter – and his story – have been preserved in a virtual installation. Using a highly realistic digital display and a state-of-the-art speech recognition system, people can now talk to “e-Gutter” as if the man were sitting in front of them. Ask him what he ate in the concentration camps, and he’ll look at you as he describes the “dregs” and “rotten meat” he was forced to consume. Ask him what he did immediately after his liberation, and Gutter will recount the day he woke in the camp to find the gates open and the guards gone. He’ll recall seeing an abandoned wagon with two horses, having the courage to climb up, and riding away from the devastation.

For decades, we have attempted to preserve people in history in interactive ways. In the late 1980s, an installation entitled Ask the President was set up at the Nixon Presidential Library. Visitors could choose from more than 280 predefined questions on a computer screen and, using archival footage or interviews filmed specifically for the project, Richard Nixon would respond appropriately. But, unlike virtual Gutter, the computerised Nixon couldn’t recognise human speech. And the breadth and depth of Gutter’s 1700 potential responses is thought to be unprecedented.

Earlier this year, the virtual Gutter was displayed in the Illinois Holocaust Museum and Education Centre. “As soon as I saw it, I was blown away,” says Amanda Friedeman, a youth educator at the centre. “He’s so accessible and so charismatic that he draws you in.” Unlike the experience of passively watching a documentary, Friedeman says actively asking questions of virtual Gutter adds “a richness and a depth of humanity”.

Over the phone, Traum plays me a passage of virtual Gutter speaking, where he talks about his last day at the Skarżysko camp in Poland. It was “being liquidated”, the voice says. The camp’s commander was choosing “people to die”, and at one point, on a whim, shot a mother and daughter. The sound quality is such that it certainly feels like Gutter himself is on the phone with me. I find I want to keep him on the line and ask questions.

Behind his affecting tone, though, is prerecorded, algorithmic trickery. In March last year, the real Gutter spent a week answering questions that the public would be likely to ask him. Meanwhile, high-definition cameras, spaced every six degrees across a 180-degree arc, encircled him, capturing his movements in remarkable detail. Gutter delivered more than 1400 responses to the various questions. But further testing revealed this was not enough to carry on natural conversations with audiences. A few months later, the octogenarian headed into the studio once more, and several hundred more clips were made. Now the system includes prerecorded responses about Gutter’s entire life, from times before, during and after the war, as well as his regrets, philosophies and even his favourite music. At one point, with a cheeky grin, Gutter told the cameras: “I have no comment about rap music.” This year, Traum’s team reported that virtual Gutter now has answers to more than 95 per cent of the questions people tend to ask him.

The next problem was getting the virtual Gutter to understand spoken queries in order for it to find the right answer in its bank of responses. “It’s a very hard problem,” says Dr Kallirroi Georgila, a colleague of Traum’s. The system first converts questions into text using a commercially available system from Google, similar to that used in smartphones. Despite gripes people have with this sort of technology, the work of Traum’s team shows that in Gutter’s case, Google correctly identifies about 95 per cent of spoken words. It tends to fail on names or places. For example, in one test “Hello Pinchas” was understood by Google as “Hello Pinterest”.

The questions are served to an algorithm that predicts the most appropriate answer from the prerecorded responses. It’s able to do this because behind the scenes Traum’s team fastidiously matched 10,000 potential queries or statements to Gutter’s best prerecorded answer. They fed this massive data set into a computer program, which picked up patterns in words and phrases in the questions and answers. The final result allows the virtual Gutter to pick out good answers to questions it has not previously been asked.

Using data collected from the Illinois Holocaust Museum, Traum and Georgila reported earlier this year that the virtual Gutter gave the best possible prerecorded response about two-thirds of the time. According to the team, this is sufficient to have “relatively open-ended conversations”. That’s probably because even if the system doesn’t give the top answer available, it often tells stories related to the question or that are “interesting in their own right”.

A small study of children who met with a survivor in person as well as virtual Gutter showed no “meaningful” differences between their experiences. At the end of a 50-minute session with e-Gutter – albeit having their questions restated by a member of the Holocaust museum, for clarity – students still had their hands raised, itching with more questions. Some waved goodbye to the image on screen. The results were presented at the International Conference on Interactive Digital Storytelling in Copenhagen this week.

In the future, Traum’s team hopes to improve the system by adding bespoke speech-recognition software to handle unique words with which Google struggles. Traum also wants virtual Gutter to consider the context of a query, rather than only isolated utterances. Currently, if someone follows up a question with “Is he still alive?”, the system would struggle to know to whom they were referring.

Virtual Gutter’s appearance will soon gain a new dimension, too. Already a 3D prototype of Gutter sits in the lab, and Traum hopes the level of detail in which Gutter was recorded will “futureproof” him, enabling the survivor to be exhibited using whatever display technology may arrive in coming years.

Since filming Gutter, the team has recorded three more survivors and is planning to expand the project. Traum believes that in time we may all be creating virtual effigies of our loved ones. “The cameras on your phone could record a sick grandfather, so that future generations could have an interactive dialogue with him,” he says. Georgila proposes a virtual companion for the elderly. “If you don’t have anyone to talk to, you could talk to a dialogue system,” she says.

Not everyone is comfortable with this. Dr Karen Sobel-Lojeski, of Stony Brook University, New York, describes the idea of virtual people, in particular digitised Holocaust survivors, as “very disturbing”. While she hasn’t yet experienced the Gutter system, Sobel-Lojeski argues that something important is “lost about our humanness when we are translated through a machine”.

Although in Traum’s mind the system “is at least as real as the contents of a diary, or a photograph”, Sobel-Lojeski says, “We have to touch diaries, and we have to be in touch with the reality of that situation. There is nothing about this system that is real.”

She is also concerned that virtual Gutter tricks people into believing they are truly conversing with someone. It’s “a fool’s game”, she says.

For Professor Martin Westwell, of Flinders University, however, the new technology should be embraced. Books, audio recordings and television documentaries each fail to capture someone’s key essence, be it their body language or minor non-verbal cues, he says. But this new technology allows us to go “one step further” in faithfully documenting our ancestors.

For Amanda Friedeman, at Illinois Holocaust Museum, there is only regret that they hadn’t produced virtual versions of several survivors who have recently died. “It would have been wonderful to capture their stories like this,” she says. “Nothing will be the same as speaking to a survivor. But when that is no longer possible, this will be a worthy substitute.”