Limitations and Trends: Audio AI LLMs: (part 3)

Author's Note: This is the third article in a series that documents changes in the Audio-LLM ("AI") landscape. It takes the form of a research log intended to capture a rough snapshot of ongoing developments. It is not my intention to delve into the ethical implications of these tools. In the future, I plan to dive deeper into the topics below. Stay tuned! 🔊

Part 1. Pink Beatles in a Purple Zeppelin
Part 2. Unnatural Selection
Part 3. Resilient

This article is divided into four sections.

1. Music LLM Demo
2. Speech Translation & Dubbing
3. Model Comparisons (Music & Sfx)
4. Limitations of LLM Tech & Trends

A list of the LLM Tools featured in this article.

1. Music Production

While the most surprising discovery comes in the next section (dubbing and translation), music "AI" provides a compelling glimpse at where the technology is generally headed for Sfx production.

After experimenting with many audio-generation LLMs for this article, I felt it would be fitting to start this section by just making the AI do it. I asked ChatGPT 3.5 to provide lyrics for a song and to suggest a genre, considering that the song may be an 'opportunity to address members of the public that might not be familiar with LLM technology'. It requested a Gospel/Blues genre.

I input this data into Suno3. Below is the song it created which uses Chat GPT's lyrics and genre request. Suno calls it Hymn of Hope: Voices of Redemption.

To provide a broad overview for the capabilities of music generating LLMs, I compiled a track using the same lyrics but only changing the genre. The track below jumps sequentially through: Epic Dance Pop, Soul, Appalachian, Contemporary Pop/Rock, Dreamy Rock, Emo 90s Rock, and Funk.

In the following section, I have included examples of ElevenLabs' AI's language translation capabilities, Below that, I provide output comparisons for Music and Sound Design LLM tools.

2. Speech Translation and Dubbing

Translation and Text to Speech tools have been improving rapidly. ElevenLabs is one of many such LLM companies focused on language processing.

Below, I requested another model from ElevenLabs to translate a simple text-based limerick into multiple languages.

Author's Note: Though I'm in the subtitle camp, I have a strong attachment to various human-dubbed anime voices. For example, YuYu Hakusho in Português is as important to me as the English voices of Goku and Vegeta. AI tech is here and jobs and creative workflows are already shifting. Voice acting is dear to me and quite different from visual acting, but that's a topic for another article.

3. Model Comparisons (Music and Sfx).

The Meat and Potatoes of this article. Below, I fed the same prompt (such as "Sapience" or "1300s Fugue Harpsichord And Electro Pop Fusion" to multiple Music and Sfx-producing LLM tools for side-by-side comparison.

LLM development for Sound Design is complicated. A multitude of tools now integrate 'Learning Models' to mix up, re-arrange, sample, and synthesize audio. SonicCharge's Snyplant2 is one of many that come to mind. However, tackling a thorough comparison of the more sophisticated sound design tools must be postponed for now. It's worth noting that the price range for these tools can be quite steep, ranging from $100 to $800.

Prompt: Sapience

Prompt: An Explosion Preceded By A Synth Bass Drop

Prompt: Horror Minimal Ambient Electronica

Prompt: Warp Engine Ignition

Prompt: 1300s Fugue Harpsichord And Electro Pop Fusion

The Technology for music is more developed than for Sfx; however, Sound Effect generation doesn't lag far behind. From my first articles to now, I have a vague feeling that Sfx generating LLMs are 6-12 months behind the music tools.

4) The technology has inherent Limitations.

Sound Design and Music hinge on the emotional and psychological resonance of sound. The arts require emotional finesse and are not merely about technical prowess or filling in auditory gaps. LLMs may one day, but do not currently, have the emotional intelligence needed to put audio to picture while matching narrative and emotional contexts.

4a. Control and Constraint.

Control surfaces are vital in art and production and Audio LLMs are currently another degree of distance from an artist's control. For example, in music, a cellist's bow gliding across strings or the combination of keys and breath control in woodwinds provide musicians with various 'surfaces' or buttons and means for controlling and expressing their music.

Controlling digitally sampled instruments or other tools via mouse and keyboard can restrict expression. Midi instruments can bridge this gap by allowing, for example, a keyboard or 3D touchpad (like the Sensel morph) to be mapped to various creative control parameters. Despite their precision and infinite customization, digital tools can still struggle to replicate the organic feel and responsiveness of traditional analog instruments.

LLM tools don't have these control surfaces. They currently function as black boxes that output audio based on text prompts or as a VST inside of Digital Audio Workstations (DAWs) which manipulate audio files based on a few mysterious digital parameters. These parameters can often be mapped to midi tools, but the degree of control is still often lacking. The control surfaces are fewer, making it difficult to get a nuanced performance.

Foley is the process of creating audio to picture. An artist might watch a scene while crinkling paper to match what's on screen and then walk in similar sounding shoes to match an actor's movement and capture emotional nuance while recording the sounds. This process results in an incredible amount of fidelity when crafting audio, and it requires highly trained artists.

I have a library with many terabytes of high quality sound effects, which I sift through when designing the audio for a project. And yet, many times I will record an analog instrument like a guitar or engage in Foley before importing a sound into my DAW because I just need something specific. I then tweak and clean and edit and crush and warp the audio in order to create a sound that fits the moment and supports the narrative of my project. This methodology grants sound designers an incredible degree of control through an extensive array of control surfaces throughout the design process. Ultimately, it is faster and produces better, more nuanced results than LLM tools when considering the demands of busy scenes and interactive game audio.

4b. Data Limitations

A large language model is only as good as the data it is trained on... sort of.

The data used to train the models is frequently unknown or unclear. Some models, like Tango, note limitations due to small sets of training data, which especially result in lower quality productions when producing content for novel concepts that have not been processed during training.

While the majority of LLM projects include obscure information regarding training data, some companies do list audio libraries and sources that have been used to train their models. AudioLDM was trained on BBC Sound Effects and Free To Use Sounds' all in one bundle, among other libraries.

I own the same Free To Use Sounds' all in one bundle and compared ocean and seagull audio with the model's output. I believe that due to the library containing a lot of ocean ambiences, which often include the sound of wind blowing over a mic at mixed intervals, the model produced a similar set of 'muddy' audio that mixed ocean sounds with the sound of wind blowing over a microphone.

4c. Trends & Additional Resources.

The tools are here, and aside from a vague understanding that these may drastically alter many industries, nobody is certain how or where things will go.

Outside of Audio, LLMs have some significant limitations. There are just things they cannot do or do well; this article, "What LLMs can never do" discusses more limitations in detail.

I would also like to share a podcast interview hosted by A Sound Effect with Jason Cushing, a key founder of SoundMorph. The podcast further discusses how Machine Learning may change the audio and sound design industry.

Afterword

Things are developing very quickly. I grew up in the era when the net came into being, and in a snap, like SOJ currency in Diablo II, it came and went and developed into something else entirely but also familiar (ie. Web 2.0).  The landscape is shifting and we're all here along for the ride.

I feel that Gene Wilder's Wonka captures the essence of the situation. This scene from the movie is definitely worth a watch if you have a moment and are in the headspace for Wonka poetry.

- ‘Wondrous Boat Ride’, from Willy Wonka and the Chocolate Factory (1971)

Times are wild, stay safe out there. Hope to see you around! ✌️