With digital twins, AI transforms the relationship between celebrities and fans

“Hey, I’m Digital Mark, Mark Tuan’s much better looking digital twin. Nah, I’m just joking.” This is how this avatar, which any fan of the American singer Mark Tuan would recognize instantly, introduces itself. In addition to the similarity with the real Mark’s face, the resemblance to his voice is also evident. However, it is the fact that you can have a conversation with him that is probably most striking. This — along with other digital twins from the world of entertainment and sports — is the creation of Soul Machines, a New Zealand company that uses artificial intelligence to make digital people. The Mark Tuan twin is one of their latest projects, but it is hardly the only one; they have also developed digital doppelgängers of boxer Francis Ngannou, former NBA basketball player Carmelo Anthony and golf legend Jack Nicklaus (his twin, however, is a replica of his 38-year-old self, a request made by the golfer himself, who is currently 83).

With digital twins like that of Mark Tuan, a singer and dancer known for being part of the k-pop group Got7, Soul Machines intends to take another step towards the future of relationships between celebrities and their fans. Unlike a chatbot like ChatGPT, Digital Mark can hold a conversation as if it were a video call: it listens to the questions he is asked, and even “watches” the user. If you smile in front of the screen, Digital Mark smiles too. If you get serious, his smile goes away. Of course, as soon as you start the tool, you must allow access to your device’s microphone and camera. According to the company, the data that is stored is anonymous, and the program — like any machine learning model — uses it to learn and improve, in addition to interpreting the user’s emotions and expressions in order to provide appropriate responses.

It only took the company four hours to bring Mark Tuan’s twin to life. In that time, they performed a 3D scan of the singer and captured as much data as possible, such as his facial movements, the texture of his skin (and his tattoos) and the way he expresses his emotions. “If you look closely at Digital Mark you can see every hair, every pore. It’s a very precise 3D model,” explains Greg Cross, CEO of Soul Machines, who founded the company with Mark Sagar, the winner of two scientific and engineering awards from the Academy of Motion Picture Arts and Sciences for his work, among other films, in the first installment of Avatar.

From there, without the need to have the artist present, the team created his voice using various recordings from sources like interviews and podcasts. Thanks to the speech generative artificial intelligence (AI), he can speak any language, such as Spanish, Korean or Thai. However, at the moment this digital clone speaks mostly English; in another language, like Spanish, he has a strong accent. In writing, the text appears in whatever language one writes, although it takes much longer.

To answer, Digital Mark uses GPT-3, a neural network machine learning model developed by OpenAI. This model is trained using information from the internet to generate any kind of text; this way, he answers any question as his flesh-and-blood counterpart would. If asked about anything regarding the singer’s life, he will provide that information in the first person. When asked, for example, if he will get back with his group, Got7, which is currently inactive, he replies: “We are still part of Got7. We haven’t left. We are all busy with our individual activities, but we will get together again when we are all available.”

The CEO of the company explains via video call from New Zealand that if you talk to Digital Mark your conversation will be different from those of other users, since the issues discussed and the questions asked will be different. However, sometimes similar answers will be given to similar questions. They can change, but if it you repeat it several times, you will notice that the variety is not infinite. Furthermore, even though he can talk about many issues, when asked about certain topics he just says he does not know anything about them. “Every time Mark has a conversation, the artificial intelligence system that brings him to life learns from the experience. That’s what’s fascinating about artificial intelligence,” says Greg Cross.

Large-scale one-on-one interactions are what Cross sees as most interesting about the future of fan relations. Once a digital twin like this has been created, it can hold tens of thousands of conversations at the same time, so anyone who wants to interact with it can do so. “When Mark announced he had a digital twin, tens of thousands of fans spoke to Digital Mark, and many came back again and again. All celebrities have a super fan base that follows everything they do, so this is another way for them to get involved with their community,” explains the CEO. Reactions among Mark Tuan’s fans seem to have been generally positive, as seen on social media. Although some do find his clone a bit strange, others claim to enjoy talking to it and share their conversations with Digital Mark on different platforms.

Just like Digital Mark, Francis Ngannou’s digital twin speaks — with his French accent — about his career and hobbies, he smiles if you smile and answers diplomatically when asked about his favorite fight: “Each one has taught me something new.” Meanwhile, Jack Nicklaus’ copy is more focused on teaching about his figure and his achievements, and his conversation topics are limited: his childhood, his charity work, his family, the 1962 US Open. As he speaks, he shows photographs to illustrate his story. The responses, being more limited, are quicker, and unlike other bots, he also asks the users questions about their conversation and doesn’t continue with the story until he gets an answer. The company is also working on a digital version of Marilyn Monroe, imagining what she would be like in the 21st century.

Although the video call format is more compelling, all the conversations include a dialogue box where the transcription of everything that has been discussed appears, and where questions can be written instead of being asked out loud. This tool is especially useful when a person has an accent or cannot speak loudly enough.

Soul Machines offers the possibility of creating digital personas for a multitude of fields, not just entertainment, celebrities and their followers. With a tool called Digital DNA Studio, any user can create a digital person and choose their hair, ethnicity, age, makeup style and more. Then, after linking it to a source of knowledge like ChatGPT, they can be added to a website. This type of program is useful in digital commerce, to answer the questions of customers and users, but this tool also makes it possible to create something like a digital influencer for social media.

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition