Spanish orchestra performs work made with AI for the first time: ‘It was a pastiche’

“Tell it to make a work in the style of [Dmitri] Shostakovich, with strength, vivacity… with metal instruments and percussion, with many dynamic contrasts,” suggests Rocío León, a member of the Symphony Orchestra at RTVE, a state-owned corporation in Spain that manages public radio and television. She’s requesting an artificial intelligence (AI) system to create music from text.

After a few minutes, the machine returns a melody. Something very metallic begins to sound. Suddenly, there’s silence: the tone changes. “It sounds like a village orchestra,” scoffs César Peña, a member of the RTVE Innovation Laboratory. “It understands the heavy metal genre, you can turn it off now,” he adds. “Do you have the essence of Shostakovich?” asks Isabella Rocafull, another member of the lab. “No,” the chatbot answers, without hesitation.

This request was just one more attempt to use AI to create a work for the Symphony Orchestra. So far, the result — just a few minutes long — has not been good.

León tries another route: “Tell it to create a piece like how Bach would write a piece for a symphony orchestra in D major for some action cartoons,” she proposes. The result is also subpar. “I don’t like it. It doesn’t have a clear structure, which is fundamental in Bach. There was a moment after the intro that was crazy. It seemed like it started well, but then it got tangled up again,” León sighs.

02:22

The work composed by AI

The objective of the RTVE Symphony Orchestra and Innovation Laboratory was to perform — for the first time — a symphonic piece composed by AI. EL PAÍS accompanied various stages of the process and spoke with several of those involved. The final two pieces were performed at the beginning of a concert that was scheduled at the Monumental Theater in Madrid, on November 17, 2023. “For the first time in the history of this theater, we’re going to start with a small experiment… and afterward, we’ll reveal the mystery,” announced Borja Arias, the conductor, while addressing the audience. He also directed the development of the two pieces. The first was transcribed exactly as it had been created by the AI, while the second was rearranged to make more sense.

At the end of the performance, Arias revealed the secret: “The two works you just heard have been composed by artificial intelligence,” he acknowledged. The audience then uttered a long “ooooh.” They were surprised by the novelty, but they were not in awe of the work of art.

Music with emojis

RTVE has documented the process — known as “Lab Orchestra” — and has been sharing it with audiences on television, radio, social media and online. The lab looked at various tools to create music with AI. Only one supported elaborate text requests, but it gave irregular results. Other tools available only allow users to describe the request with labels or emojis, before offering something less elaborate in return.

Since 2022, AI has been a creative tool for texts and images, with ChatGPT getting the most attention, along with other graphic tools, such as Dall-E or Midjourney. Musicians have seen this process up close, but in their sector, they haven’t experienced such a big shakeup. “What we’re going to ask of the machine is to construct — to the extent that it can — a musical composition,” explained Manuel Ventero — managing director of the RTVE Symphony Orchestra — at the beginning of the process. “We sense that the result will be, in the worst case, very good,” he added.

But the complexity of classical music poses an even more complicated challenge for the machine: it’s one thing to fill in the background music of a YouTube video and another to create a work that’s performed by dozens of musicians. The process showed that, for now, AI doesn’t have the capabilities of Mozart or Bach, or even that of a dedicated student of composition. “It puts out very amateurish things,” Arias laments. They also tried an AI tool to transcribe music. It didn’t work.

“This sounds like Indiana Jones”

The requests (or prompts) from those in charge of RTVE’s orchestra and lab were initially elaborate, as is the norm when asking AI for detailed illustrations. The proposals aspired for the machine to combine composers, genres and instruments. But the AI produced messy sounds, like an unruly puzzle. The first day of testing — which took place last September, on the stage of the Monumental Theater — already indicated that it wasn’t going to be easy: “This sounds more like Indiana Jones,” “there are reggaeton influences here,” or “it doesn’t understand this language” were some of the comments from that session.

After about 30 tests, those in charge of the project settled on a very standard request to obtain an acceptable result: “A piece of music to be played by a symphony orchestra, created by a human composer.” The music was mediocre, but the composers let it be. Clearly, the AI won’t be taking away their jobs… for now.

Human labor remains indispensable. Borja Arias received about 20 minutes of music created by AI, collected in 10 cuts. “I tried to take what the machine wanted to tell me; I took few elements, because there were many,” he explains. This is how Arias compares what he received from the machine with other arts: “Musical coherence is abstract and difficult to achieve. When you compose, you have to stick to a style. This piece had many inputs and there was no cohesion. It was as if it started to tell you a story about a protagonist who’s a firefighter… and then suddenly continued telling you a story about a carpenter, a flower girl and a judge,” he describes.

The RTVE project ensured that the musicians in the orchestra wouldn’t know who the composer was until the end. They rehearsed and performed the piece as if it were anonymous. Members of the lab interviewed members of the orchestra before they learned the truth. The musicians knew that something was off, but they couldn’t pinpoint what it was. “I didn’t like the first work [created by AI and only transcribed] at all… but, well, it’s also true that we’ve only read it once,” admitted trumpet player Borja Antón after the first rehearsal. “Perhaps with a little more reading and work, we can make it take on a little more shape. But it feels sort of like a basic work.”

When he was finally told who had composed the piece, he wasn’t very surprised. “Quite a few things struck me. The first piece was very bad. We couldn’t do anything with the concept. In the second [which has been rearranged by Arias], the melody was catchier, the structure was better,” he added.

Useful for inspiration

To the artists’ expert ears, it was obvious that there were things that were going wrong: “If it doesn’t sound familiar to you, it might seem like a hodgepodge of movie music that then tries to be more classical and then contemporary, but without being innovative. It’s a pastiche. It wasn’t unpleasant to listen to, it’s true. I imagined that the musicians were going to think it was by a mediocre composer,” León affirms.

For some reason, AI has a harder time creating something musically meaningful for an orchestra. “There are things that artificial intelligence doesn’t know are very difficult to play,” Antón opines. “There’s a high C that goes on for a long time — that’s impossible to do. Physically, it’s not possible. And then, there are no different planes, it’s all in one go: you hear the melody with the harmony all together and you don’t recognize anything.”

The difficulty of classical music doesn’t have to be repeated in other styles. “To compose symphonic music, the truth is that AI has a long way to go,” Arias affirms. “This experience has left me feeling quite calm: it has demonstrated the value that this type of music has, due to its harmonic complexity and fairly profound message. With other styles, the AI is a little closer to achieving optimal results. I’m peaceful knowing that there’s still room for improvement.”

This lack of complexity, however, doesn’t prevent AI from also being a useful tool to inspire composers in their work, or to imagine less sophisticated works, according to León: “Composers could take it into account as a little bit of help, so that they don’t have to face the blank page. For a film, AI can serve as a fill-in musical moment, as an atmosphere. But not for an orchestra, which is something deeper and richer.”

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition